Azure Kubernetes Service (AKS)<\/strong> is a highly available, secure, and fully managed Kubernetes<\/a> <\/span>service of Microsoft Azure. Terraform is a tool for building, changing, and versioning infrastructure safely and efficiently. Terraform can manage existing and popular service providers as well as custom in-house solutions.<\/p>\n\n\n\n Admitting, there are number of Cloud provisioning IaC tools, each with its own implementation. With the object of this lab, We will focus exclusively on deploying AKS cluster on Azure.<\/p>\n\n\n\n This guide walks you through how to the following tasks:<\/p>\n\n\n\n Using the Azure Portal you can create a cluster with few clicks. \u2b50\ufe0fInfrastructure as Code, in simple terms, is a means by which we can write declarative definitions for the infrastructure we want to exist and using them with a provisioning tool that deals with the actual deployment. The first step is to obtain the source code from my Github repository. git clone https:\/\/github.com\/AymenSegni\/azure-aks-k8s-tf.git The directory that holds the Terraform configuration files for this lab has a special tree structure. 1- With Terraform, you can put a bunch of code inside of a Every Terraform configuration has at least one module, known as its root module<\/em> (the In this lab we have a well defined structure of the TF Modules. Let\u2019s go through the The module is a container for multiple resources that are used together.<\/p>\n\n\n\n In this stage, will see together a full example of the aks cluster module.<\/p>\n\n\n\n Resources<\/em> are the most important element in the Terraform language. Each resource block describes one or more infrastructure objects, such as virtual networks, compute instances, or higher-level components such as DNS records.<\/p>\n<\/div><\/div>\n\n\n\n variable cluster_name { Input variables serve as parameters for a Terraform module, allowing aspects of the module to be customized without altering the module’s own source code, and allowing modules to be shared between different configurations.<\/p>\n\n\n\n output “azurerm_kubernetes_cluster_id” { Output values are like the return values of a Terraform module.<\/p>\n\n\n\n Well, we have discovered what\u2019s the main components of a Terraform module. Now, is a good time to call this function -module- in the root deployment, as well as all the other Terraform modules that define the AKS cluster<\/p>\n\n\n\n The root module -deployment- contains also tow other Terraform files: A provider configuration is created using a provider “azurerm” { The name given in the block header ( Each time a new provider is added to configuration — either explicitly via a Provider initialization is one of the actions of A Terraform backend storage ( terraform { In the second part of this lab will set up Azure storage to store Terraform state.<\/p>\n\n\n\n Once you\u2019ve declared all your resources, defined your necessary variables and provided your credentials, you\u2019re all set to let Terraform do its magic! To check if everything will work and there\u2019s no errors, run Are you still exited to run your Kubernetes cluster on Cloud ? \ud83d\ude2c Terraform tracks state locally via the In this section, we will see how to do the following tasks:<\/p>\n\n\n\n Using your terminal, create a container in your Azure storage account. Replace the placeholders with appropriate values for your environment.<\/p>\n\n\n\n In this section, you see how to use the 1. In your local terminal, initialize Terraform. Replace the placeholders with appropriate values for your environment<\/p>\n\n\n\n The 2. Export your service principal credentials. Replace the placeholders with appropriate values from your service principal<\/p>\n\n\n\n\n 3. Run the The 4. Run the To manage a Kubernetes cluster, you use\u00a0kubectl<\/a>, the Kubernetes command-line client. If you use Azure Cloud Shell,\u00a0 To configure\u00a0 To verify the connection to your cluster, use the\u00a0kubectl get<\/a>\u00a0command to return a list of the cluster nodes.<\/p>\n\n\n\n You should see the details of your worker nodes, and they should all have a status Ready<\/strong>, as shown in the following example:<\/p>\n\n\n\n When the AKS cluster was created, monitoring was enabled to capture health metrics for both the cluster nodes and pods. These health metrics are available in the Azure portal. For more information on container health monitoring, see Monitor Azure Kubernetes Service healt<\/a><\/span>h<\/span><\/a>.<\/p>\n\n\n\n If you make changes to your code, running the The setup described is only the beginning, if you’re provisioning production-grade infrastructure you should look into:<\/p>\n\n\n\n
The fully managed Azure Kubernetes Service (AKS) makes deploying and managing containerized applications easy. It offers serverless Kubernetes, an integrated continuous integration and continuous delivery (CI\/CD) experience, and enterprise-grade security and governance. Unite your development and operations teams on a single platform to rapidly build, deliver, and scale applications with confidence.<\/p>\n\n\n\nObjectives<\/h2>\n\n\n\n
Assumptions and Prerequisites<\/h2>\n\n\n\n
Azure<\/a><\/code><\/span><\/li>Kubernetes<\/code><\/li>Terraform<\/code> installed in your local machine<\/li>Software Dependencies<\/h2>\n\n\n\n
Why use Terraform (or any other IaC tool) to create an AKS cluster ?<\/h2>\n\n\n\n
However, it usually a better idea to keep the configuration for your cluster under version control. Assuming you accidentally delete your cluster or decide to provision a copy in another region, you can easily replicate the same configuration. And if you’re working as part of a team, source control gives you peace of mind. You know precisely why changes occurred and who made them.<\/p>\n\n\n\n
This means that we can code what we want built, provide necessary credentials for the given provider, kick off the provisioning process, pop the kettle on and come back to find all your services purring along nicely in the cloud\u2026 or a terminal screen full of ominous warnings about failed deployment and \u201cunrecoverable state\u201d and a deep sense of growing unease (but not often, don\u2019t worry \ud83d\udc4d).<\/p>\n<\/div><\/div>\n\n\n\n![\"Using](\"http:\/\/aymen-segni.com\/wp-content\/uploads\/2019\/12\/IoC_twitterpost01_dc3eac7bbbbe620b391cff9aaebf16f3.jpg\")
<\/figure><\/div>\n\n\n\n
\n\n\n\nDefine AKS cluster with Terraform<\/h2>\n\n\n\n
This will clone the sample repository and make it the current directory:<\/p>\n\n\n\n
cd azure-aks-k8s-tf<\/p>\n\n\n\n
Obviously, there are 2 main subfolders: deployment<\/strong><\/code> and modules<\/code><\/strong>. In order to see the source code structure you can run tree <\/em><\/code>
<\/p>\n\n\n\n![\"Tree](\"https:\/\/aymen-segni.com\/wp-content\/uploads\/2020\/04\/Screenshot-2020-04-05-at-17.00.05-1024x734.png\")
<\/figure>\n<\/div><\/div>\n<\/div><\/div>\n\n\n\nProject structure <\/h4>\n\n\n\n
modules<\/code><\/strong>: represent here in this layout the Terraform modules (general re-used functions) . In this lab, we have basically 4 modules:
– aks_cluste<\/code>r: the main unit providing the AKS service
– aks_identities<\/code>: the cluster identity unit that manage the cluster service principal
– aks_network:<\/code> Create the cluster Virtual Network and subnetwork on Azure
– <\/strong>log_analytics<\/code>: Formally Azure Operational Insight is the unit that manages logs and cluster health checks2- Deployment<\/code>: <\/strong> is the main function of this layout, responsible of the AKS Kubernetes cluster deployment on Azure.
In main.tf<\/code><\/strong> we define the Terraform modules already created in \/modules<\/code> sub-folder with the appropriate inputs defined in variables.tf<\/code><\/strong> or in a terraform.tfvars<\/code> file (wich is not covered in this guide).<\/p>\n\n\n\n
Stay tuned, in the next section, we\u2019re going to talk about how to create reusable infrastructure with Terraform.<\/p>\n\n\n\nTerraform modules<\/h3>\n\n\n\n
Terraform module<\/code> and reuse that module in multiple places throughout your code. Instead of having the same code copy\/pasted in the staging and production environments<\/code>, you\u2019ll be able to have both environments reuse code from the same module.
This is a big deal. Modules are the key ingredient to writing reusable, maintainable, and testable Terraform code.<\/p>\n\n\n\n \/deployment<\/code> in this lab context), which consists of the resources defined in the .tf<\/code> files in the main working directory.<\/p>\n\n\n\nTerraform module structure<\/strong><\/h4>\n\n\n\n
aks_identities<\/code> module as an example:<\/p>\n\n\n\n![\"Create](\"https:\/\/aymen-segni.com\/wp-content\/uploads\/2020\/04\/Screenshot-2020-04-05-at-17.10.55-1024x246.png\")
<\/figure>\n\n\n\nmain.tf:<\/code> the aks cluster resources are packaged in the main.tf file <\/li>variables.tf<\/code>: In Terraform, modules can have input parameters, too. To define them, you use a mechanism input variables.<\/li>output.tf<\/code>: In Terraform, a module can also return values. Again, this is done using a mechanism: output variables.<\/li><\/ol>\n\n\n\n![\"Now,](\"http:\/\/aymen-segni.com\/wp-content\/uploads\/2019\/12\/i-know-tf-module.jpeg\")
resource \"azurerm_kubernetes_cluster\" \"cluster\" {\n name = var.cluster_name\n location = var.location\n resource_group_name = var.resource_group_name\n dns_prefix = var.dns_prefix\n kubernetes_version = var.kubernetes_version\n\n default_node_pool {\n name = var.default_pool_name\n node_count = var.node_count\n vm_size = var.vm_size\n os_disk_size_gb = var.os_disk_size_gb\n vnet_subnet_id = var.vnet_subnet_id\n max_pods = var.max_pods\n type = var.default_pool_type\n\n enable_auto_scaling = true\n min_count = var.min_count\n max_count = var.max_count\n\n tags = merge(\n {\n \"environment\" = \"runitoncloud\"\n },\n {\n \"aadssh\" = \"True\"\n },\n )\n }\n\n\n network_profile {\n network_plugin = var.network_plugin\n network_policy = \"calico\"\n service_cidr = var.service_cidr\n dns_service_ip = \"10.0.0.10\"\n docker_bridge_cidr = \"172.17.0.1\/16\"\n }\n\n service_principal {\n client_id = var.client_id\n client_secret = var.client_secret\n }\n\n\n tags = {\n Environment = \"Development\"\n }\n\n lifecycle {\n prevent_destroy = true\n }\n}\n\nresource \"azurerm_monitor_diagnostic_setting\" \"aks_cluster\" {\n name = \"${azurerm_kubernetes_cluster.cluster.name}-audit\"\n target_resource_id = azurerm_kubernetes_cluster.cluster.id\n log_analytics_workspace_id = var.diagnostics_workspace_id\n\n log {\n category = \"kube-apiserver\"\n enabled = true\n\n retention_policy {\n enabled = false\n }\n }\n\n log {\n category = \"kube-controller-manager\"\n enabled = true\n\n retention_policy {\n enabled = false\n }\n }\n\n log {\n category = \"cluster-autoscaler\"\n enabled = true\n\n retention_policy {\n enabled = false\n }\n }\n\n log {\n category = \"kube-scheduler\"\n enabled = true\n\n retention_policy {\n enabled = false\n }\n }\n\n log {\n category = \"kube-audit\"\n enabled = true\n\n retention_policy {\n enabled = false\n }\n }\n\n metric {\n category = \"AllMetrics\"\n enabled = false\n\n retention_policy {\n enabled = false\n }\n }\n}\n<\/code><\/pre>\n\n\n\nTerraform resources<\/strong><\/h4>\n\n\n\n
resource \"azurerm_kubernetes_cluster\" \"cluster\" {}<\/code>
This block is responsible for creating the AKS cluster<\/p>\n\n\n\nvariable \"dns_prefix\" {\n description = \"DNS prefix\"\n}\nvariable \"location\" {\n description = \"azure location to deploy resources\"\n}\nvariable \"cluster_name\" {\n description = \"AKS cluster name\"\n}\nvariable \"resource_group_name\" {\n description = \"name of the resource group to deploy AKS cluster in\"\n}\nvariable \"kubernetes_version\" {\n description = \"version of the kubernetes cluster\"\n}\nvariable \"api_server_authorized_ip_ranges\" {\n description = \"ip ranges to lock down access to kubernetes api server\"\n default = \"0.0.0.0\/0\"\n}\n# Node Pool config\nvariable \"agent_pool_name\" {\n description = \"name for the agent pool profile\"\n default = \"default\"\n}\nvariable \"agent_pool_type\" {\n description = \"type of the agent pool (AvailabilitySet and VirtualMachineScaleSets)\"\n default = \"AvailabilitySet\"\n}\nvariable \"node_count\" {\n description = \"number of nodes to deploy\"\n}\nvariable \"vm_size\" {\n description = \"size\/type of VM to use for nodes\"\n}\nvariable \"os_disk_size_gb\" {\n description = \"size of the OS disk to attach to the nodes\"\n}\nvariable \"vnet_subnet_id\" {\n description = \"vnet id where the nodes will be deployed\"\n}\nvariable \"max_pods\" {\n description = \"maximum number of pods that can run on a single node\"\n}\n\n#Network Profile config\nvariable \"network_plugin\" {\n description = \"network plugin for kubenretes network overlay (azure or calico)\"\n default = \"azure\"\n}\nvariable \"service_cidr\" {\n description = \"kubernetes internal service cidr range\"\n default = \"10.0.0.0\/16\"\n}\nvariable \"diagnostics_workspace_id\" {\n description = \"log analytics workspace id for cluster audit\"\n}\nvariable \"min_count\" {\n default = 1\n description = \"Minimum Node Count\"\n}\nvariable \"max_count\" {\n default = 5\n description = \"Maximum Node Count\"\n}\nvariable \"default_pool_name\" {\n description = \"name for the agent pool profile\"\n default = \"default\"\n}\nvariable \"default_pool_type\" {\n description = \"type of the agent pool (AvailabilitySet and VirtualMachineScaleSets)\"\n default = \"VirtualMachineScaleSets\"\n}\nvariable \"client_id\" { \n}\nvariable \"client_secret\" {\n}<\/code><\/pre>\n\n\n\nTerraform variabels<\/strong><\/h4>\n\n\n\n
description = “AKS cluster name”
default = “run-it-on-cloud”
}<\/p>\n\n\n\noutput \"azurerm_kubernetes_cluster_id\" {\n value = azurerm_kubernetes_cluster.cluster.id\n}\n\noutput \"azurerm_kubernetes_cluster_fqdn\" {\n value = azurerm_kubernetes_cluster.cluster.fqdn\n}\n\noutput \"azurerm_kubernetes_cluster_node_resource_group\" {\n value = azurerm_kubernetes_cluster.cluster.node_resource_group\n}<\/code><\/pre>\n\n\n\nTerraform output<\/strong><\/h4>\n\n\n\n
value = azurerm_kubernetes_cluster.cluster.id
}<\/p>\n\n\n\n# Cluster Resource Group\nresource \"azurerm_resource_group\" \"aks\" {\n name = var.resource_group_name\n location = var.location\n}\n\n# AKS Cluster Network\nmodule \"aks_network\" {\n source = \"..\/modules\/aks_network\"\n subnet_name = var.subnet_name\n vnet_name = var.vnet_name\n resource_group_name = azurerm_resource_group.aks.name\n subnet_cidr = var.subnet_cidr\n location = var.location\n address_space = var.address_space\n}\n\n# AKS IDs\nmodule \"aks_identities\" {\n source = \"..\/modules\/aks_identities\"\n cluster_name = var.cluster_name\n}\n\n# AKS Log Analytics\nmodule \"log_analytics\" {\n source = \"..\/modules\/log_analytics\"\n resource_group_name = azurerm_resource_group.aks.name\n log_analytics_workspace_location = var.log_analytics_workspace_location\n log_analytics_workspace_name = var.log_analytics_workspace_name\n log_analytics_workspace_sku = var.log_analytics_workspace_sku\n}\n\n# AKS Cluster\nmodule \"aks_cluster\" {\n source = \"..\/modules\/aks-cluster\"\n cluster_name = var.cluster_name\n location = var.location\n dns_prefix = var.dns_prefix\n resource_group_name = azurerm_resource_group.aks.name\n kubernetes_version = var.kubernetes_version\n node_count = var.node_count\n min_count = var.min_count\n max_count = var.max_count\n os_disk_size_gb = \"1028\"\n max_pods = \"110\"\n vm_size = var.vm_size\n vnet_subnet_id = module.aks_network.aks_subnet_id\n client_id = module.aks_identities.cluster_client_id\n client_secret = module.aks_identities.cluster_sp_secret\n diagnostics_workspace_id = module.log_analytics.azurerm_log_analytics_workspace\n}<\/code><\/pre>\n\n\n\n Define the cluster deployment<\/h3>\n\n\n\n
– As you can see, we start building the cluster by defining a cluster resource group. You can learn more about Azure resource group here<\/a><\/span>.
– Next, we create the AKS Cluster Network by create the cluster Vnet and subnet using the TF network module already defined in the modules.
– The next block is for the AKS IDs. learn more about Azure SPs here<\/span>
<\/a>– Now everything is ready to create the AKS cluster.
– The last section of this deployment consists of logs and the cluster health checks <\/p>\n<\/div><\/div>\n\n\n\n
<\/p>\n\n\n\nprovider<\/code><\/strong>.<\/code>tf<\/code><\/strong>: While r<\/span><\/a>esources<\/a><\/span> are the primary construct in the Terraform language, the behaviors<\/em> of resources rely on their associated resource types, and these types are defined by providers<\/em>.
Learn more about Terraform provider h<\/span><\/a>ere<\/a><\/span>
In the next section will discuss the project provider configuration.<\/li><\/ul>\n\n\n\nterraform {\n backend \"azurerm\" {}\n}\n\nprovider \"azurerm\" {\n version = \"~> 2.4\"\n features {}\n}<\/code><\/pre>\n\n\n\nprovider<\/code> block:<\/p>\n\n\n\n
version = “~>2.4”
}<\/p>\n\n\n\nazurem Cloud provider)<\/code> is the name of the provider to configure. Terraform associates each resource type with a provider by taking the first word of the resource type name<\/p>\n\n\n\nprovider<\/code> block or by adding a resource from that provider — Terraform must initialize the provider before it can be used. Initialization downloads and installs the provider’s plugin so that it can later be executed.<\/p>\n\n\n\nInitialization<\/strong><\/h4>\n\n\n\n
terraform init<\/code>.
Running this command will download and initialize any providers that are not already initialized.<\/p>\n\n\n\n\n
terraform init
<\/code><\/pre>\n\n\n\n\n![\"Terraform](\"http:\/\/aymen-segni.com\/wp-content\/uploads\/2019\/12\/tf-init-1024x787.png\")
terraform state<\/code> ) is create using a terraform black:<\/p>\n\n\n\n
backend “azurerm” {}
}<\/p>\n\n\n\nrequired_version<\/code> setting can be used to constrain which versions of the Terraform CLI can be used with your configuration.
Note<\/strong>: that all the lab code are written for Terraform 0.12.x.<\/code><\/li><\/ul>\n\n\n\nterraform validate<\/code> and terraform plan<\/code> from within the directory.
If all is well and you\u2019re happy with what it plans to build, kick off the process with terraform apply<\/code>, one final approval, then wait for your infrastructure to be deployed \ud83d\ude42 <\/p>\n\n\n\n
So, let Terraform do its magic!<\/p>\n\n\n\n![\"Terraform](\"http:\/\/aymen-segni.com\/wp-content\/uploads\/2019\/12\/tf-apply-1.jpg\")
Set up Azure storage to store Terraform state<\/h2>\n\n\n\n
terraform.tfstate<\/code> file.
This pattern works well in a single-person environment.
However, in a multi-person environment, A<\/a>zure storage<\/a> <\/span>is used to track the tf state<\/code>.<\/p>\n\n\n\nAll services <\/code>in the left menu.<\/li>Storage accounts<\/code>.<\/li>Storage accounts <\/code>tab, select the name of the storage account into which Terraform is to store state. For example, you can use the storage account created when you opened Cloud Shell the first time. The storage account name created by Cloud Shell typically starts with cs<\/code> followed by a random string of numbers and letters. Take note of the storage account you select. This value is needed later.<\/li>Access keys.<\/code><\/li> key1 key<\/code> value. (Selecting the icon to the right of the key copies the value to the clipboard.)<\/li><\/ol>\n\n\n\n![\"Terraform](\"http:\/\/aymen-segni.com\/wp-content\/uploads\/2019\/12\/Screenshot-2019-12-23-at-21.49.05-1024x478.png\")
![\"terraform](\"http:\/\/aymen-segni.com\/wp-content\/uploads\/2019\/12\/Screenshot-2019-12-23-at-21.50.49-1024x361.png\")
az storage container create -n tfstate --account-name <YourAzureStorageAccountName> --account-key \\<YourAzureStorageAccountKey><\/code><\/pre>\n\n\n\nCreate AKS using Terraform<\/h2>\n\n\n\n
terraform init<\/code> command to create the resources defined the configuration files you created in the previous sections.<\/p>\n\n\n\ncd src\/deployment\nterraform init -backend-config=\"storage_account_name=<YourAzureStorageAccountName>\" -backend-config=\"container_name=tfstate\" -backend-config=\"access_key=<YourStorageAccountAccessKey>\" -backend-config=\"key=codelab.microsoft.tfstate\"<\/code><\/pre>\n\n\n\nterraform init<\/code> command displays the success of initializing the backend and provider plug-in:<\/p>\n\n\n\n
export TF_VAR_client_id=<service-principal-appid>
<\/code><\/pre>\n\n\n\n\n\n
export TF_VAR_client_secret=<service-principal-password>
<\/code><\/pre>\n\n\n\n\nterraform plan<\/code> command to create the Terraform plan that defines the infrastructure elements.<\/p>\n\n\n\n\n
terraform plan -out out.plan
<\/code><\/pre>\n\n\n\n\nterraform plan<\/code> command displays the resources that will be created when you run the terraform apply<\/code> command.
There\u2019s an Terraform plan output example:<\/p>\n\n\n\n![\"Terraform](\"http:\/\/aymen-segni.com\/wp-content\/uploads\/2019\/12\/tf-plan-811x1024.png\")
![\"Terraform](\"http:\/\/aymen-segni.com\/wp-content\/uploads\/2019\/12\/Screenshot-2019-12-24-at-00.17.04-870x1024.png\")
terraform apply<\/code> command to apply the plan to create the Kubernetes cluster. The process to create a Kubernetes cluster can take several minutes.<\/p>\n\n\n\n\n
terraform apply out.plan
<\/code><\/pre>\n\n\n\n\n![\"\"](\"https:\/\/aymen-segni.com\/wp-content\/uploads\/2020\/07\/Screenshot-2020-07-23-at-01.51.44-1024x966.png\")
![\"\"](\"https:\/\/aymen-segni.com\/wp-content\/uploads\/2020\/07\/Screenshot-2020-07-23-at-01.54.11-1024x417.png\")
Test the Kubernetes AKS cluster<\/h2>\n\n\n\n
kubectl<\/code>\u00a0is already installed. To install\u00a0kubectl<\/code>\u00a0locally, use the\u00a0az aks install-cli<\/a>\u00a0command:<\/p>\n\n\n\naz aks install-cli<\/code><\/pre>\n\n\n\nkubectl<\/code>\u00a0to connect to your Kubernetes cluster, use the\u00a0az aks get-credentials<\/a>\u00a0command. This command downloads credentials and configures the Kubernetes CLI to use them. (make sure to use your own created resource group and AKS names)<\/p>\n\n\n\naz aks get-credentials --resource-group runitoncloud --name runItOnCloud<\/code><\/pre>\n\n\n\n|~\/run-it-on-cloud\/azure-aks-k8s-tf\/src\/deployment \ud83d\udcbb --\u2446 master \ud83d\udd27 \n|\n|--\u2af8 kubectl get nodes \nNAME STATUS ROLES AGE VERSION\naks-default-17028763-vmss000000 Ready agent 16m v1.16.10\naks-default-17028763-vmss000001 Ready agent 16m v1.16.10<\/code><\/pre>\n\n\n\nMonitor health and logs<\/h2>\n\n\n\n
Next steps<\/h2>\n\n\n\n
plan<\/code> and apply<\/code> commands again will let Terraform use its knowledge of the deployed resources (.tfstate)<\/code> to calculate what changes need to be made, whether building or destroying. Finally, when you want to bring down your infrastructure, simply issue a terraform destroy<\/code> command and down it comes.<\/p>\n\n\n\n