38 C
Monday, July 22, 2024

Deploy Apache Kafka in Azure HDInsight using reusable Terraform modules

We will deploy the resources below using Terraform reusable modules utilizing the Azure landing zone concept, part of the Cloud Adoption Framework (CAF). In this setup, we are talking about only infra resources; if you are new to terraform, the same concept has been explained using the Azure Portal; I have spoken about Azure Management Groups and Subscription Planning in this link – Azure Management Groups and Subscriptions Design.

Azure landing zone design that accounts for scale, security governance, networking, and identity, which enables seamless application migration, modernization, and innovation at the enterprise scale in Azure. This approach considers all platform resources like infrastructure (Iaas) or platform as a service.

Benefits of Azure Landing Zones –

  • Good Governance ( Like you can place a policy in the overall environment that no internet-exposing storage accounts can be provisioned)
  • Security (Improved Security controls, Network segmentation, Identity management, Service Principals, Managed Identities)
  • Scalability (Multi Datacenter or Improving the design with Virtual WAN should be seamless)
  • Cost Savings (Segregated billing with subscriptions – Overall Control or like can apply Hybrid benefit using policies)
  • Resources Provisioned –

1.  Virtual Networks ( Hub – – Spoke –
2. VPN Gateway ( 
3.  Apache Kafka in Azure HDInsight
4.  Public IP Addresses

Head Nodes “”
Worker Nodes = “”
Zone Keeper Nodse = “”

Modules are convenient to place into folders and reuse resource configurations with Terraform for multiple deployments.
Also, changing/upgrading specific resource configurations becomes easier.

Git Hub Repo link –

azure365pro/azure-hub-spoke-terraform (github.com)

Needed Resources –

  • Terraform’s latest version is installed.
terraform -version
  • az CLI is installed / az login is completed
az login
  • Git is installed to clone the repository

Let’s clone the repository

git clone "https://github.com/azure365pro/azure-hub-spoke-terraform"

Get inside the repo

cd .\azure-hub-spoke-terraform\


main-kafka-example.tf to main-kafka.tf
uncomment the main-kafka.tf file


Run Terraform init to create a local tfstate

terraform init

Make sure you are running on the right subscription. If you have access to multiple subscriptions

terraform plan

Now terraform apply command is used to create the resources.

terraform apply

Now terraform destroy command is used to clean up the resources.

terraform destroy

Running the same with Azure DevOps Releases with Muti Stage Approvals

We utilize stages, triggers, approvers, and deployment options in release pipelines.

Verify Stage with run init / plan

Prod Stage with run init/plan / apply

Satheshwaran Manoharan
Satheshwaran Manoharanhttps://www.azure365pro.com
Award-winning Technology Leader with a wealth of experience running large teams and diversified industry exposure in cloud computing. From shipping lines to rolling stocks.In-depth expertise in driving cloud adoption strategies and modernizing systems to cloud native. Specialized in Microsoft Cloud, DevOps, and Microsoft 365 Stack and conducted numerous successful projects worldwide. Also, Acting as a Technical Advisor for various start-ups.

Related Articles


Please enter your comment!
Please enter your name here

× How can I help you?