In this article, we will explore the different approaches to save cloud costs with Kubernetes on Azure but it will apply in (nearly) exactly the same way for other cloud providers.<\/p>\n\n\n\n
Assumptions and Prerequisites<\/h3>\n\n\n\n\n- Basic hands-on experience with Kubernetes.<\/li>\n\n\n\n
- Up and Running Azure AKS Kubernetes Cluster.<\/li>\n\n\n\n
- You have kubectl<\/a> \u00a0installed in your local machine.<\/li>\n\n\n\n
- You have Azure CLI<\/a> installed.<\/li>\n\n\n\n
- Preferably you have Terraform<\/a> installed in your local machine<\/li>\n\n\n\n
- Running the Azure CLI version 2.0.76 or later. Run\u00a0
az --version<\/code>\u00a0to find the version. If you need to install or upgrade, see\u00a0Install Azure CLI<\/a>.<\/li>\n<\/ul>\n\n\n\nHow to save cloud costs with Kubernetes on Azure ?<\/h3>\n\n\n\n
To answer this question, we’ll cover the following techniques and tools in this article:<\/p>\n\n\n\n
\n- \u2705 Enable and use the cluster autoscaler in an AKS cluster<\/li>\n\n\n\n
- \u2705 Scale down your cluster(s) during non-work hours<\/li>\n\n\n\n
- \u2705 Use Spot instances<\/li>\n\n\n\n
- \u2705 How to horizontally autoscale your K8S deployments (HPA)<\/li>\n\n\n\n
- \u2705 Clean up unused resources on your cluster<\/li>\n\n\n\n
- \u2705 Optimise your Kubernetes resource allocation<\/li>\n<\/ol>\n\n\n\n
Before starting<\/h3>\n\n\n\n
To create an Azure AKS cluster, preferably you use an Infrastructure as Code tool like Terraform. Although cluster creation isn\u2019t covered by this lab, however you can follow my IaC lab to deploy AKS with Terraform<\/a>.<\/p>\n\n\n\n![\"Save](\"https:\/\/aymen-segni.com\/wp-content\/uploads\/2020\/12\/Screenshot-2020-12-13-at-15.13.45-1024x520.png\")
Azure AKS Cluster<\/figcaption><\/figure>\n\n\n\nEnable and use the cluster autoscaler in an AKS cluster<\/h3>\n\n\n\n
To keep up with application demands in Azure Kubernetes Service (AKS), you may need to adjust the number of nodes that run your workloads. The cluster autoscaler component can watch for pods in your cluster that can’t be scheduled because of resource constraints. When issues are detected, the number of nodes in a node pool is increased to meet the application demand. Nodes are also regularly checked for a lack of running pods, with the number of nodes then decreased as needed. This ability to automatically scale up or down the number of nodes in your AKS cluster lets you run an efficient, cost-effective cluster.<\/p>\n\n\n\n
Update an existing AKS cluster to enable the cluster autoscaler<\/h4>\n\n\n\n
Use the az aks update<\/a> command to enable and configure the cluster autoscaler on the node pool for the existing cluster. Use the --enable-cluster-autoscaler<\/code> parameter, and specify a node --min-count<\/code> and --max-count<\/code>.<\/p>\n\n\n\n Important<\/strong>:<\/p>\n\n\n\nThe cluster autoscaler is a Kubernetes component. Although the AKS cluster uses a virtual machine scale set for the nodes, don’t manually enable or edit settings for scale set autoscale in the Azure portal or using the Azure CLI. Let the Kubernetes cluster autoscaler manage the required scale settings. For more information, see Can I modify the AKS resources in the node resource group?<\/a><\/p>\n\n\n\nThe following example updates an existing AKS cluster to enable the cluster autoscaler on the node pool for the cluster and sets a minimum of 1<\/em> and maximum of 3<\/em> nodes:Azure CLICopyTry It<\/p>\n\n\n\naz aks update \\\n --resource-group <yourResourceGroup> \\\n --name <yourAKSCluster> \\\n --enable-cluster-autoscaler \\\n --min-count 1 \\\n --max-count 10<\/code><\/pre>\n\n\n\nIt takes a few minutes to update the cluster and configure the cluster autoscaler settings.<\/p>\n\n\n\n
Scale down your cluster during non-work hours<\/h3>\n\n\n\n
Kubernetes Downscaler<\/a> tool (kube-downscaler) allows users and operators to scale down systems during off-hours. To save Cloud costs with Kubernetes on Azure you need to combine the downscaler with the AKS cluster autoscaler.<\/p>\n\n\n\nKubernetes deployments and StatefulSets can be scaled to zero replicas. CronJobs can be suspended. Kubernetes Downscaler is configured for the whole cluster, one or more namespaces, or individual resources. Either “downtime” or its inverse, “uptime”, can be set. <\/p>\n\n\n\n
The following showing you my AKS cluster’s nodes -only 1 node- before I deploy anything on it:<\/p>\n\n\n\n
|--\u2af8 kubectl get nodes NAME \n STATUS ROLES AGE VERSION\n aks-default-17028763-vmss000000 Ready agent 9d v1.19.3<\/code><\/pre>\n\n\n\nNow let\u2019s deploy the Online Boutique<\/a><\/strong> micro-services demo application on AKS and watch the behaviour of the cluster<\/p>\n\n\n\ngit clone https:\/\/github.com\/GoogleCloudPlatform\/microservices-demo.git <\/code>\ncd microservices-demo<\/code>\nkubectl apply -f .\/release\/kubernetes-manifests.yaml <\/code><\/code><\/pre>\n\n\n\nIn order to handle the new traffic, we will see that the cluster will scale up itself in few minutes<\/p>\n\n\n\n
|--\u2af8 kubectl get nodes \n NAME STATUS ROLES AGE VERSION\n aks-default-17028763-vmss000000 Ready agent 9d v1.19.3\n aks-default-17028763-vmss000002 Ready agent 2m29s v1.19.3<\/code><\/pre>\n\n\n\nAnd now we can check the status of our application deployment:<\/p>\n\n\n\n
|--\u2af8 kubectl get pods \n NAME READY STATUS RESTARTS AGE\n adservice-5f6f7c76f5-25vzj 1\/1 Running 0 8m1s\n cartservice-675b6659c8-jhgrq 1\/1 Running 5 8m2s\n checkoutservice-85d4b74f95-6wg28 1\/1 Running 0 8m3s\n currencyservice-6d7f8fc9fc-dtjq9 1\/1 Running 0 8m2s\n emailservice-798f4f5575-ptpjm 1\/1 Running 0 8m3s\n frontend-6b64dc9665-9vstg 1\/1 Running 0 8m2s\n loadgenerator-7747b67b5-gcwlm 1\/1 Running 6 8m2s\n paymentservice-98cb47fff-b9qxf 1\/1 Running 0 8m2s\n productcatalogservice-7f857c47f-6kdmh 1\/1 Running 0 8m2s\n recommendationservice-5bf5bcbbdf-6mfpd 1\/1 Running 0 8m2s\n redis-cart-74594bd569-clrf9 1\/1 Running 0 8m1s\n shippingservice-75f7f9dc6c-c5vmc 1\/1 Running 0 8m1s<\/code><\/pre>\n\n\n\n In the direction of demonstrating the power of the cluster autoscaler and kube-downscaler combined, w’ll start by deploying and configuring the kube-downscaler operator in our cluster, in fact, this guy will shrink the application deployment immediately – as it\u2019s a non-working hour at the moment of writing this article \ud83d\ude09 –<\/p>\n\n\n\n
https:\/\/codeberg.org\/hjacobs\/kube-downscaler.git\ncd kube-downscaler<\/code><\/pre>\n\n\n\nThen, we will edit the configuration file: deploy\/deployment.yaml as the following: <\/p>\n\n\n\n
image: hjacobs\/kube-downscaler:20.10.0\n args:\n # dry run by default, remove to perform downscaling\n # - --dry-run\n # run every minute\n - --interval=30\n - --exclude-namespaces=kube-system\n - --exclude-deployments=kube-downscaler\n - --default-uptime=Mon-Fri 08:00-20:00 Europe\/Berlin\n - --include-resources=deployments,statefulsets,stacks,cronjobs\n - --deployment-time-annotation=deployment-time<\/code><\/pre>\n\n\n\nNow we rae ready to run our operator:<\/p>\n\n\n\n
kubectl apply -f deploy\/ <\/code><\/code><\/pre>\n\n\n\nAs expected, the Kube Downscaler scale down everything we deployed during the shift to 0 except the downscaler pod itself \ud83d\ude03<\/p>\n\n\n\n
|--\u2af8 kubectl get pods \n NAME READY STATUS RESTARTS AGE\n kube-downscaler-77cffd57fb-xjkpc 1\/1 Running 0 168m<\/code><\/pre>\n\n\n\nMoreover, we can see that there is only one node left in our cluster, thanks to the magic magic of the AKS Cluster autoscaler \ud83d\ude48<\/p>\n\n\n\n
|--\u2af8 kubectl get nodes \n NAME STATUS ROLES AGE VERSION\n aks-default-17028763-vmss000000 Ready agent 9d v1.19.3<\/code><\/pre>\n\n\n\nIn production, this tool helped us to scale down from ~13 to 3 worker nodes and this certainly makes a difference on the our Azure bill.<\/p>\n\n\n\n
Add a spot node pool to an AKS cluster<\/h3>\n\n\n\n
Next practice, Azure AKS costs can be reduced by using Spot instances as Kubernetes worker nodes. Spot Instances are available at up to a 90% discount compared to On-Demand prices.<\/p>\n\n\n\n
A spot node pool is a node pool backed by a spot virtual machine scale set<\/a>. Using spot VMs for nodes with your AKS cluster allows you to take advantage of unutilized capacity in Azure at a significant cost savings.<\/p>\n\n\n\nImportant 1: When deploying a spot node pool, Azure will allocate the spot nodes if there’s capacity available. But there’s no SLA for the spot nodes. A spot scale set that backs the spot node pool is deployed in a single fault domain and offers no high availability guarantees. At any time when Azure needs the capacity back, the Azure infrastructure will evict spot nodes.<\/p>\n\n\n\n
Important 2: Spot nodes are great for workloads that can handle interruptions, early terminations, or evictions. For example, workloads such as batch processing jobs, development and testing environments, and large compute workloads may be good candidates to be scheduled on a spot node pool.<\/p>\n\n\n\n
In this article, we will add a secondary spot node pool to an existing Azure Kubernetes Service (AKS) cluster.
Create a node pool using the az aks nodepool add<\/a>:
<\/p>\n\n\n\naz aks nodepool add \\\n--resource-group <yourResourceGroup> \\\n--cluster-name <yourAKSCluster> \\\n--name spotnodepool \\\n--priority Spot \\\n--eviction-policy Delete \\\n--spot-max-price -1 \\\n--min-count 1 \\\n--max-count 10 \\\n--no-wait<\/code><\/pre>\n\n\n\nBy default, you create a node pool with a priority<\/em> of Regular<\/em> in your AKS cluster when you create a cluster with multiple node pools. The above command adds an auxiliary node pool to an existing AKS cluster with a priority<\/em> of Spot<\/em>. The priority<\/em> of Spot<\/em> makes the node pool a spot node pool. The eviction-policy<\/em> parameter is set to Delete<\/em> in the above example, which is the default value. When you set the eviction policy<\/a> to Delete<\/em>, nodes in the underlying scale set of the node pool are deleted when they’re evicted. You can also set the eviction policy to Deallocate<\/em>. When you set the eviction policy to Deallocate<\/em>, nodes in the underlying scale set are set to the stopped-deallocated state upon eviction. Nodes in the stopped-deallocated state count against your compute quota and can cause issues with cluster scaling or upgrading. The priority<\/em> and eviction-policy<\/em> values can only be set during node pool creation. Those values can’t be updated later.<\/p>\n\n\n\nVerify the spot node pool<\/h4>\n\n\n\n
To verify your node pool has been added as a spot node pool:Azure CLICopy<\/p>\n\n\n\n
az aks nodepool show --resource-group <yourResourceGroup> --cluster-name <yourAKSCluster> --name spotnodepool\n<\/code><\/pre>\n\n\n\nConfirm scaleSetPriority<\/em> is Spot<\/em>.<\/p>\n\n\n\nTo schedule a pod to run on a spot node, add a toleration that corresponds to the taint applied to your spot node. The following example shows a portion of a yaml file that defines a toleration that corresponds to a kubernetes.azure.com\/scalesetpriority=spot:NoSchedule<\/em> taint used in the previous step.YAMLCopy<\/p>\n\n\n\nspec:\n containers:\n - name: spot-example\n tolerations:\n - key: \"kubernetes.azure.com\/scalesetpriority\"\n operator: \"Equal\"\n value: \"spot\"\n effect: \"NoSchedule\"\n ...\n<\/code><\/pre>\n\n\n\nWhen a pod with this toleration is deployed, Kubernetes can successfully schedule the pod on the nodes with the taint applied.<\/p>\n\n\n\n
Max price for a spot pool<\/h4>\n\n\n\n
Pricing for spot instances is variable<\/a>, based on region and SKU. For more information, see pricing for Linux<\/a> and Windows<\/a>.<\/p>\n\n\n\nWith variable pricing, you have option to set a max price, in US dollars (USD), using up to 5 decimal places. For example, the value 0.98765<\/em> would be a max price of $0.98765 USD per hour. If you set the max price to -1<\/em>, the instance won’t be evicted based on price. The price for the instance will be the current price for Spot or the price for a standard instance, whichever is less, as long as there is capacity and quota available.<\/p>\n\n\n\nLimitations<\/h4>\n\n\n\n
The following limitations apply when you create and manage AKS clusters with a spot node pool:<\/p>\n\n\n\n
\n- A spot node pool can’t be the cluster’s default node pool. A spot node pool can only be used for a secondary pool.<\/li>\n\n\n\n
- You can’t upgrade a spot node pool since spot node pools can’t guarantee cordon and drain. You must replace your existing spot node pool with a new one to do operations such as upgrading the Kubernetes version. To replace a spot node pool, create a new spot node pool with a different version of Kubernetes, wait until its status is\u00a0Ready<\/em>, then remove the old node pool.<\/li>\n\n\n\n
- The control plane and node pools cannot be upgraded at the same time. You must upgrade them separately or remove the spot node pool to upgrade the control plane and remaining node pools at the same time.<\/li>\n\n\n\n
- A spot node pool must use Virtual Machine Scale Sets.<\/li>\n\n\n\n
- You cannot change ScaleSetPriority or SpotMaxPrice after creation.<\/li>\n\n\n\n
- When setting SpotMaxPrice, the value must be -1 or a positive value with up to five decimal places.<\/li>\n\n\n\n
- A spot node pool will have the label\u00a0kubernetes.azure.com\/scalesetpriority:spot<\/em>, the taint\u00a0kubernetes.azure.com\/scalesetpriority=spot:NoSchedule<\/em>, and system pods will have anti-affinity.<\/li>\n\n\n\n
- You must add a\u00a0corresponding toleration<\/a>\u00a0to schedule workloads on a spot node pool.<\/li>\n<\/ul>\n\n\n\n
Clean up unused resources<\/h3>\n\n\n\n
Working in a fast-paced environment is great. We want tech organizations to accelerate. Faster software delivery also means more PR deployments, preview environments, prototypes, and spike solutions. All deployed on Kubernetes. Who has time to clean up test deployments manually? It’s easy to forget about deleting last week’s experiment. The cloud bill will eventually grow because of things we forget to shut down<\/p>\n\n\n\n
Kubernetes Janitor<\/a> (kube-janitor) helps to clean up your cluster. The janitor configuration is flexible for both global and local usage:<\/p>\n\n\n\n\n- Generic cluster-wide rules can dictate a maximum time-to-live (TTL) for PR\/test deployments.<\/li>\n\n\n\n
- Individual resources can be annotated with\u00a0janitor\/ttl, e.g. to automatically delete a spike\/prototype after 7 days.<\/li>\n<\/ul>\n\n\n\n
Generic rules are defined in a YAML file. Its path is passed via the –rules-file option to kube-janitor. Here an example rule to delete all namespaces with -pr- in their name after two days:<\/p>\n\n\n\n
- id: cleanup-resources-from-pull-requests \nresources:\n - namespaces \njmespath: \"contains(metadata.name, '-pr-')\" \nttl: 2d<\/code><\/pre>\n\n\n\nTo require the application label on Deployment and StatefulSet Pods for all new Deployments\/StatefulSet in 2020, but still allow running tests without this label for a week:<\/p>\n\n\n\n
- id: require-application-label \n# remove deployments and statefulsets without a label \"application\"<\/em> \nresources: \n- deployments - statefulsets # see http:\/\/jmespath.org\/specification.html<\/em> \njmespath: \"!(spec.template.metadata.labels.application) && metadata.creationTimestamp > '2020-01-01'\" \nttl: 7d<\/code><\/pre>\n\n\n\nTo run a time-limited demo for 30 minutes in a cluster where kube-janitor is running:<\/p>\n\n\n\n
kubectl run nginx-demo --image=nginx\nkubectl annotate deploy nginx-demo janitor\/ttl=30m\n<\/code><\/pre>\n\n\n\nAnother source of growing costs are persistent volumes (Azure ABS). Deleting a Kubernetes StatefulSet will not delete its persistent volumes (PVCs). Unused ABS volumes can easily cause costs of hundreds of dollars per month. Kubernetes Janitor has a feature to clean up unused PVCs. For example, this rule will delete all PVCs which are not mounted by a Pod and not referenced by a StatefulSet or CronJob:<\/p>\n\n\n\n
# delete all PVCs which are not mounted and not referenced by StatefulSets<\/em> \n- id: remove-unused-pvcs \nresources: \n- persistentvolumeclaims \njmespath: \"_context.pvc_is_not_mounted && _context.pvc_is_not_referenced\" \nttl: 24h \n<\/code><\/pre>\n\n\n\nKubernetes Janitor can help you keep your cluster “clean” and prevent slowly growing cloud costs. See the kube-janitor README fo<\/a>r instructions on how to deploy and configure.<\/p>\n\n\n\nOptimise your Kubernetes resource allocation<\/h3>\n\n\n\n
Kubernetes workloads specify their CPU\/memory needs via “resource requests”. CPU resources are measured in virtual cores or more commonly in “millicores”, e.g. 500m denoting 50% of a vCPU. Memory resources are measured in Bytes and the usual suffixes can be used, e.g. 500Mi denoting 500 Mebibyte. Resource requests “block” capacity on worker nodes, i.e. a Pod with 1000m CPU requests on a node with 4 vCPUs will leave only 3 vCPUs available for other Pods. <\/p>\n\n\n\n
Slack<\/em> is the difference between the requested resources and the real usage. For example, a Pod which requests 2 GiB of memory, but only uses 200 MiB, has ~1.8 GiB of memory “slack”. Slack costs money. We can roughly say that 1 GiB of memory slack costs ~$10\/month.<\/p>\n\n\n\nKubernetes Resource Report<\/a> (kube-resource-report) displays slack and can help you identify saving potential:<\/p>\n\n\n\n![\"Kubernetes](\"https:\/\/srcco.de\/galleries\/saving-cloud-costs-kubernetes-aws\/kube-resource-report-slack.png\")
<\/figure>\n\n\n\nKubernetes Resource Report shows slack aggregated by application and team. This allows finding opportunities where resource requests can be lowered. The generated HTML report only provides a snapshot of resource usage. You need to look at CPU\/memory usage over time to set the right resource requests. Here a Grafana chart of a “typical” service with a lot of CPU slack: all Pods stay well below the 3 requested CPU cores:<\/p>\n\n\n\n![\"Kubernetes](\"https:\/\/srcco.de\/galleries\/saving-cloud-costs-kubernetes-aws\/grafana-cpu-requests-usage.png\")
<\/figure>\n\n\n\nReducing the CPU requests from 3000m to ~400m frees up resources for other workloads and allows the cluster to scale down.<\/p>\n\n\n\n
“The average CPU utilization of EC2 instances often hovers in the range of single-digit percentages” writes Corey Quinn<\/a>. While EC2 Right Sizing might be wrong<\/a>, changing some Kubernetes resource requests in a YAML file is easy to do and can yield huge savings.<\/p>\n\n\n\nBut do we really want humans to change numbers in YAML files? No, machines can do this better! The Kubernetes Vertical Pod Autoscaler<\/a> (VPA) does exactly this: adapt resource requests and limits to match the workload. Here some example graph of Prometheus’ CPU requests (thin blue line) adapted by VPA over time:<\/p>\n\n\n\n![\"VPA](\"https:\/\/srcco.de\/galleries\/saving-cloud-costs-kubernetes-aws\/vpa-prometheus-cpu-requests.png\")
<\/figure>\n\n\n\nZalando uses the VPA in all its clusters<\/a> for infrastructure components. Non-critical applications can also use VPA.<\/p>\n\n\n\nFairwind’s Goldilocks<\/a> is a tool creating a VPA for each deployment in a namespace and then shows the VPA recommendation on its dashboard. This can help developers set the right CPU\/memory requests for their application(s):<\/p>\n\n\n\n![\"Goldilocks](\"https:\/\/srcco.de\/galleries\/saving-cloud-costs-kubernetes-aws\/goldilocks-example.png\")
<\/figure>\n\n\n\nUse horizontal autoscaling<\/h3>\n\n\n\n
Last, but not least, will introduce the HPA to save cloud costs with Kubernetes on Azure. Many applications\/services deal with a dynamic load pattern: sometimes their Pods idle and sometimes they are at their capacity. Running always with a fleet of Pods to handle the maximum peak load is not cost efficient. Kubernetes supports horizontal autoscaling via the HorizontalPodAutoscaler<\/a> (HPA) resource. CPU usage is often a good metric to scale on:<\/p>\n\n\n\napiVersion: autoscaling\/v2beta2\nkind: HorizontalPodAutoscaler\nmetadata:\n name: my-app\nspec:\n scaleTargetRef:\n apiVersion: apps\/v1\n kind: Deployment\n name: my-app\n minReplicas: 3\n maxReplicas: 10\n metrics:\n - type: Resource\n resource:\n name: cpu\n target:\n averageUtilization: 100\n type: Utilization\n <\/code><\/pre>\n\n\n\nConfiguring horizontal autoscaling with HPA should be one of the default actions to increase efficiency for stateless services. Spotify has a presentation with their learnings and recommendations for HPA: Scale Your Deployments, Not Your Wallet<\/a>.<\/p>\n\n\n\nThat\u2019s all folks!<\/h3>\n\n\n\n
That\u2019s all for this lab, thanks for reading \ud83d\ude4f
I hope you find some of the presented tools useful to reduce your cloud bill.<\/p>\n\n\n\n
References and credits<\/h3>\n\n\n\n\n- Featured image: Caylent<\/a><\/li>\n\n\n\n
- https:\/\/srcco.de\/posts\/saving-cloud-costs-kubernetes-aws.html<\/a><\/li>\n\n\n\n
- https:\/\/docs.microsoft.com\/en-us\/azure\/aks\/spot-node-pool<\/a><\/li>\n\n\n\n
- https:\/\/docs.microsoft.com\/en-us\/azure\/aks\/cluster-autoscaler<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"
This blog describe how you can save Cloud costs with Kubernetes on Azure while running at best performance and reliability.<\/p>\n","protected":false},"author":1,"featured_media":813,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[1],"tags":[],"class_list":["post-810","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-non-classe"],"yoast_head":"\n
Save cloud costs with Kubernetes on Azure<\/title>\n<meta name=\"description\" content=\"This blog describe how you can save Cloud costs with Kubernetes on Azure while running at best performance and reliability.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/aymen-segni.com\/index.php\/2020\/12\/23\/save-cloud-costs-with-kubernetes-on-azure\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Save cloud costs with Kubernetes on Azure\" \/>\n<meta property=\"og:description\" content=\"This blog describe how you can save Cloud costs with Kubernetes on Azure while running at best performance and reliability.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/aymen-segni.com\/index.php\/2020\/12\/23\/save-cloud-costs-with-kubernetes-on-azure\/\" \/>\n<meta property=\"og:site_name\" content=\"Run It On Cloud\" \/>\n<meta property=\"article:published_time\" content=\"2020-12-23T00:22:36+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-06-10T07:29:44+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/aymen-segni.com\/wp-content\/uploads\/2020\/12\/improve_cost.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"671\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"aymen-segni\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@https:\/\/x.com\/axsegni\" \/>\n<meta name=\"twitter:site\" content=\"@axsegni\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"aymen-segni\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"15 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/aymen-segni.com\/index.php\/2020\/12\/23\/save-cloud-costs-with-kubernetes-on-azure\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/aymen-segni.com\/index.php\/2020\/12\/23\/save-cloud-costs-with-kubernetes-on-azure\/\"},\"author\":{\"name\":\"aymen-segni\",\"@id\":\"https:\/\/aymen-segni.com\/#\/schema\/person\/32033966e7bd410bbaf1b79c7e94b59d\"},\"headline\":\"Save cloud costs with Kubernetes on Azure\",\"datePublished\":\"2020-12-23T00:22:36+00:00\",\"dateModified\":\"2024-06-10T07:29:44+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/aymen-segni.com\/index.php\/2020\/12\/23\/save-cloud-costs-with-kubernetes-on-azure\/\"},\"wordCount\":2557,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/aymen-segni.com\/#\/schema\/person\/32033966e7bd410bbaf1b79c7e94b59d\"},\"articleSection\":[\"Non class\u00e9\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/aymen-segni.com\/index.php\/2020\/12\/23\/save-cloud-costs-with-kubernetes-on-azure\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/aymen-segni.com\/index.php\/2020\/12\/23\/save-cloud-costs-with-kubernetes-on-azure\/\",\"url\":\"https:\/\/aymen-segni.com\/index.php\/2020\/12\/23\/save-cloud-costs-with-kubernetes-on-azure\/\",\"name\":\"Save cloud costs with Kubernetes on Azure\",\"isPartOf\":{\"@id\":\"https:\/\/aymen-segni.com\/#website\"},\"datePublished\":\"2020-12-23T00:22:36+00:00\",\"dateModified\":\"2024-06-10T07:29:44+00:00\",\"description\":\"This blog describe how you can save Cloud costs with Kubernetes on Azure while running at best performance and reliability.\",\"breadcrumb\":{\"@id\":\"https:\/\/aymen-segni.com\/index.php\/2020\/12\/23\/save-cloud-costs-with-kubernetes-on-azure\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/aymen-segni.com\/index.php\/2020\/12\/23\/save-cloud-costs-with-kubernetes-on-azure\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/aymen-segni.com\/index.php\/2020\/12\/23\/save-cloud-costs-with-kubernetes-on-azure\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Accueil\",\"item\":\"https:\/\/aymen-segni.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Save cloud costs with Kubernetes on Azure\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/aymen-segni.com\/#website\",\"url\":\"https:\/\/aymen-segni.com\/\",\"name\":\"Run It On Cloud\",\"description\":\"Accelerate your Cloud & MLOps Journey\",\"publisher\":{\"@id\":\"https:\/\/aymen-segni.com\/#\/schema\/person\/32033966e7bd410bbaf1b79c7e94b59d\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/aymen-segni.com\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":[\"Person\",\"Organization\"],\"@id\":\"https:\/\/aymen-segni.com\/#\/schema\/person\/32033966e7bd410bbaf1b79c7e94b59d\",\"name\":\"aymen-segni\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/aymen-segni.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/aymen-segni.com\/wp-content\/uploads\/2025\/02\/72799.jpg\",\"contentUrl\":\"https:\/\/aymen-segni.com\/wp-content\/uploads\/2025\/02\/72799.jpg\",\"width\":896,\"height\":1152,\"caption\":\"aymen-segni\"},\"logo\":{\"@id\":\"https:\/\/aymen-segni.com\/#\/schema\/person\/image\/\"},\"description\":\"Staff Engineer with over a decade of experience in building, scaling, and leading MLOPS, Cloud Native, SRE, and DevOps platforms across high-growth and enterprise environments. I specialize in architecting production-grade systems with a strong emphasis on resilience, security, and developer experience; bringing together deep expertise in distributed systems, Kubernetes, and modern platform engineering to empower engineering teams and accelerate business value. My work spans Cloud (AWS, GCP, Azure, OpenStack), Kubernetes, SRE (SLOs, observability, incident response), AI infrastructure and AgentOps (vLLM, Nvidia, RayServe, etc), and Platform Engineering (Backstage, Keptn, GitOps, self-service). I\u2019ve led teams through Cloud Native transformations, established scalable SRE practices, and built internal platforms that streamline operations and reduce cognitive load. With a strong programming background, and Infrastructure as Code (Terraform, Helm, Ansible), I drive automation-first approaches to eliminate toil, ensure reliability, and enable secure, compliant deployment pipelines. My focus today is on building Cloud Native AI platforms, where DevOps meets AI Infrastructure Stacks to support scalable, production-ready LLMs and AI Platforms. As a dedicated mentor, both within my teams and through platforms like MentorCruise, I am passionate about helping engineers perform at their best and assisting organizations in scaling with confidence. Driven by systems thinking, platform-as-a-product mindset, and engineering excellence, I help teams ship faster, operate smarter, and scale with confidence.\",\"sameAs\":[\"https:\/\/aymen-segni.com\",\"https:\/\/www.linkedin.com\/in\/aymen-segni\",\"https:\/\/twitter.com\/https:\/\/x.com\/axsegni\"],\"url\":\"https:\/\/aymen-segni.com\/index.php\/author\/admin8647\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Save cloud costs with Kubernetes on Azure","description":"This blog describe how you can save Cloud costs with Kubernetes on Azure while running at best performance and reliability.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/aymen-segni.com\/index.php\/2020\/12\/23\/save-cloud-costs-with-kubernetes-on-azure\/","og_locale":"en_US","og_type":"article","og_title":"Save cloud costs with Kubernetes on Azure","og_description":"This blog describe how you can save Cloud costs with Kubernetes on Azure while running at best performance and reliability.","og_url":"https:\/\/aymen-segni.com\/index.php\/2020\/12\/23\/save-cloud-costs-with-kubernetes-on-azure\/","og_site_name":"Run It On Cloud","article_published_time":"2020-12-23T00:22:36+00:00","article_modified_time":"2024-06-10T07:29:44+00:00","og_image":[{"width":1200,"height":671,"url":"https:\/\/aymen-segni.com\/wp-content\/uploads\/2020\/12\/improve_cost.png","type":"image\/png"}],"author":"aymen-segni","twitter_card":"summary_large_image","twitter_creator":"@https:\/\/x.com\/axsegni","twitter_site":"@axsegni","twitter_misc":{"Written by":"aymen-segni","Est. reading time":"15 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/aymen-segni.com\/index.php\/2020\/12\/23\/save-cloud-costs-with-kubernetes-on-azure\/#article","isPartOf":{"@id":"https:\/\/aymen-segni.com\/index.php\/2020\/12\/23\/save-cloud-costs-with-kubernetes-on-azure\/"},"author":{"name":"aymen-segni","@id":"https:\/\/aymen-segni.com\/#\/schema\/person\/32033966e7bd410bbaf1b79c7e94b59d"},"headline":"Save cloud costs with Kubernetes on Azure","datePublished":"2020-12-23T00:22:36+00:00","dateModified":"2024-06-10T07:29:44+00:00","mainEntityOfPage":{"@id":"https:\/\/aymen-segni.com\/index.php\/2020\/12\/23\/save-cloud-costs-with-kubernetes-on-azure\/"},"wordCount":2557,"commentCount":0,"publisher":{"@id":"https:\/\/aymen-segni.com\/#\/schema\/person\/32033966e7bd410bbaf1b79c7e94b59d"},"articleSection":["Non class\u00e9"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/aymen-segni.com\/index.php\/2020\/12\/23\/save-cloud-costs-with-kubernetes-on-azure\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/aymen-segni.com\/index.php\/2020\/12\/23\/save-cloud-costs-with-kubernetes-on-azure\/","url":"https:\/\/aymen-segni.com\/index.php\/2020\/12\/23\/save-cloud-costs-with-kubernetes-on-azure\/","name":"Save cloud costs with Kubernetes on Azure","isPartOf":{"@id":"https:\/\/aymen-segni.com\/#website"},"datePublished":"2020-12-23T00:22:36+00:00","dateModified":"2024-06-10T07:29:44+00:00","description":"This blog describe how you can save Cloud costs with Kubernetes on Azure while running at best performance and reliability.","breadcrumb":{"@id":"https:\/\/aymen-segni.com\/index.php\/2020\/12\/23\/save-cloud-costs-with-kubernetes-on-azure\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/aymen-segni.com\/index.php\/2020\/12\/23\/save-cloud-costs-with-kubernetes-on-azure\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/aymen-segni.com\/index.php\/2020\/12\/23\/save-cloud-costs-with-kubernetes-on-azure\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Accueil","item":"https:\/\/aymen-segni.com\/"},{"@type":"ListItem","position":2,"name":"Save cloud costs with Kubernetes on Azure"}]},{"@type":"WebSite","@id":"https:\/\/aymen-segni.com\/#website","url":"https:\/\/aymen-segni.com\/","name":"Run It On Cloud","description":"Accelerate your Cloud & MLOps Journey","publisher":{"@id":"https:\/\/aymen-segni.com\/#\/schema\/person\/32033966e7bd410bbaf1b79c7e94b59d"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/aymen-segni.com\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":["Person","Organization"],"@id":"https:\/\/aymen-segni.com\/#\/schema\/person\/32033966e7bd410bbaf1b79c7e94b59d","name":"aymen-segni","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/aymen-segni.com\/#\/schema\/person\/image\/","url":"https:\/\/aymen-segni.com\/wp-content\/uploads\/2025\/02\/72799.jpg","contentUrl":"https:\/\/aymen-segni.com\/wp-content\/uploads\/2025\/02\/72799.jpg","width":896,"height":1152,"caption":"aymen-segni"},"logo":{"@id":"https:\/\/aymen-segni.com\/#\/schema\/person\/image\/"},"description":"Staff Engineer with over a decade of experience in building, scaling, and leading MLOPS, Cloud Native, SRE, and DevOps platforms across high-growth and enterprise environments. I specialize in architecting production-grade systems with a strong emphasis on resilience, security, and developer experience; bringing together deep expertise in distributed systems, Kubernetes, and modern platform engineering to empower engineering teams and accelerate business value. My work spans Cloud (AWS, GCP, Azure, OpenStack), Kubernetes, SRE (SLOs, observability, incident response), AI infrastructure and AgentOps (vLLM, Nvidia, RayServe, etc), and Platform Engineering (Backstage, Keptn, GitOps, self-service). I\u2019ve led teams through Cloud Native transformations, established scalable SRE practices, and built internal platforms that streamline operations and reduce cognitive load. With a strong programming background, and Infrastructure as Code (Terraform, Helm, Ansible), I drive automation-first approaches to eliminate toil, ensure reliability, and enable secure, compliant deployment pipelines. My focus today is on building Cloud Native AI platforms, where DevOps meets AI Infrastructure Stacks to support scalable, production-ready LLMs and AI Platforms. As a dedicated mentor, both within my teams and through platforms like MentorCruise, I am passionate about helping engineers perform at their best and assisting organizations in scaling with confidence. Driven by systems thinking, platform-as-a-product mindset, and engineering excellence, I help teams ship faster, operate smarter, and scale with confidence.","sameAs":["https:\/\/aymen-segni.com","https:\/\/www.linkedin.com\/in\/aymen-segni","https:\/\/twitter.com\/https:\/\/x.com\/axsegni"],"url":"https:\/\/aymen-segni.com\/index.php\/author\/admin8647\/"}]}},"jetpack_featured_media_url":"https:\/\/aymen-segni.com\/wp-content\/uploads\/2020\/12\/improve_cost.png","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/aymen-segni.com\/index.php\/wp-json\/wp\/v2\/posts\/810","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/aymen-segni.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/aymen-segni.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/aymen-segni.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/aymen-segni.com\/index.php\/wp-json\/wp\/v2\/comments?post=810"}],"version-history":[{"count":41,"href":"https:\/\/aymen-segni.com\/index.php\/wp-json\/wp\/v2\/posts\/810\/revisions"}],"predecessor-version":[{"id":1089,"href":"https:\/\/aymen-segni.com\/index.php\/wp-json\/wp\/v2\/posts\/810\/revisions\/1089"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/aymen-segni.com\/index.php\/wp-json\/wp\/v2\/media\/813"}],"wp:attachment":[{"href":"https:\/\/aymen-segni.com\/index.php\/wp-json\/wp\/v2\/media?parent=810"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/aymen-segni.com\/index.php\/wp-json\/wp\/v2\/categories?post=810"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/aymen-segni.com\/index.php\/wp-json\/wp\/v2\/tags?post=810"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
az --version<\/code>\u00a0to find the version. If you need to install or upgrade, see\u00a0Install Azure CLI<\/a>.<\/li>\n<\/ul>\n\n\n\nHow to save cloud costs with Kubernetes on Azure ?<\/h3>\n\n\n\n
To answer this question, we’ll cover the following techniques and tools in this article:<\/p>\n\n\n\n
\n- \u2705 Enable and use the cluster autoscaler in an AKS cluster<\/li>\n\n\n\n
- \u2705 Scale down your cluster(s) during non-work hours<\/li>\n\n\n\n
- \u2705 Use Spot instances<\/li>\n\n\n\n
- \u2705 How to horizontally autoscale your K8S deployments (HPA)<\/li>\n\n\n\n
- \u2705 Clean up unused resources on your cluster<\/li>\n\n\n\n
- \u2705 Optimise your Kubernetes resource allocation<\/li>\n<\/ol>\n\n\n\n
Before starting<\/h3>\n\n\n\n
To create an Azure AKS cluster, preferably you use an Infrastructure as Code tool like Terraform. Although cluster creation isn\u2019t covered by this lab, however you can follow my IaC lab to deploy AKS with Terraform<\/a>.<\/p>\n\n\n\nAzure AKS Cluster<\/figcaption><\/figure>\n\n\n\nEnable and use the cluster autoscaler in an AKS cluster<\/h3>\n\n\n\n
To keep up with application demands in Azure Kubernetes Service (AKS), you may need to adjust the number of nodes that run your workloads. The cluster autoscaler component can watch for pods in your cluster that can’t be scheduled because of resource constraints. When issues are detected, the number of nodes in a node pool is increased to meet the application demand. Nodes are also regularly checked for a lack of running pods, with the number of nodes then decreased as needed. This ability to automatically scale up or down the number of nodes in your AKS cluster lets you run an efficient, cost-effective cluster.<\/p>\n\n\n\n
Update an existing AKS cluster to enable the cluster autoscaler<\/h4>\n\n\n\n
Use the az aks update<\/a> command to enable and configure the cluster autoscaler on the node pool for the existing cluster. Use the --enable-cluster-autoscaler<\/code> parameter, and specify a node --min-count<\/code> and --max-count<\/code>.<\/p>\n\n\n\n Important<\/strong>:<\/p>\n\n\n\nThe cluster autoscaler is a Kubernetes component. Although the AKS cluster uses a virtual machine scale set for the nodes, don’t manually enable or edit settings for scale set autoscale in the Azure portal or using the Azure CLI. Let the Kubernetes cluster autoscaler manage the required scale settings. For more information, see Can I modify the AKS resources in the node resource group?<\/a><\/p>\n\n\n\nThe following example updates an existing AKS cluster to enable the cluster autoscaler on the node pool for the cluster and sets a minimum of 1<\/em> and maximum of 3<\/em> nodes:Azure CLICopyTry It<\/p>\n\n\n\naz aks update \\\n --resource-group <yourResourceGroup> \\\n --name <yourAKSCluster> \\\n --enable-cluster-autoscaler \\\n --min-count 1 \\\n --max-count 10<\/code><\/pre>\n\n\n\nIt takes a few minutes to update the cluster and configure the cluster autoscaler settings.<\/p>\n\n\n\n
Scale down your cluster during non-work hours<\/h3>\n\n\n\n
Kubernetes Downscaler<\/a> tool (kube-downscaler) allows users and operators to scale down systems during off-hours. To save Cloud costs with Kubernetes on Azure you need to combine the downscaler with the AKS cluster autoscaler.<\/p>\n\n\n\nKubernetes deployments and StatefulSets can be scaled to zero replicas. CronJobs can be suspended. Kubernetes Downscaler is configured for the whole cluster, one or more namespaces, or individual resources. Either “downtime” or its inverse, “uptime”, can be set. <\/p>\n\n\n\n
The following showing you my AKS cluster’s nodes -only 1 node- before I deploy anything on it:<\/p>\n\n\n\n
|--\u2af8 kubectl get nodes NAME \n STATUS ROLES AGE VERSION\n aks-default-17028763-vmss000000 Ready agent 9d v1.19.3<\/code><\/pre>\n\n\n\nNow let\u2019s deploy the Online Boutique<\/a><\/strong> micro-services demo application on AKS and watch the behaviour of the cluster<\/p>\n\n\n\ngit clone https:\/\/github.com\/GoogleCloudPlatform\/microservices-demo.git <\/code>\ncd microservices-demo<\/code>\nkubectl apply -f .\/release\/kubernetes-manifests.yaml <\/code><\/code><\/pre>\n\n\n\nIn order to handle the new traffic, we will see that the cluster will scale up itself in few minutes<\/p>\n\n\n\n
|--\u2af8 kubectl get nodes \n NAME STATUS ROLES AGE VERSION\n aks-default-17028763-vmss000000 Ready agent 9d v1.19.3\n aks-default-17028763-vmss000002 Ready agent 2m29s v1.19.3<\/code><\/pre>\n\n\n\nAnd now we can check the status of our application deployment:<\/p>\n\n\n\n
|--\u2af8 kubectl get pods \n NAME READY STATUS RESTARTS AGE\n adservice-5f6f7c76f5-25vzj 1\/1 Running 0 8m1s\n cartservice-675b6659c8-jhgrq 1\/1 Running 5 8m2s\n checkoutservice-85d4b74f95-6wg28 1\/1 Running 0 8m3s\n currencyservice-6d7f8fc9fc-dtjq9 1\/1 Running 0 8m2s\n emailservice-798f4f5575-ptpjm 1\/1 Running 0 8m3s\n frontend-6b64dc9665-9vstg 1\/1 Running 0 8m2s\n loadgenerator-7747b67b5-gcwlm 1\/1 Running 6 8m2s\n paymentservice-98cb47fff-b9qxf 1\/1 Running 0 8m2s\n productcatalogservice-7f857c47f-6kdmh 1\/1 Running 0 8m2s\n recommendationservice-5bf5bcbbdf-6mfpd 1\/1 Running 0 8m2s\n redis-cart-74594bd569-clrf9 1\/1 Running 0 8m1s\n shippingservice-75f7f9dc6c-c5vmc 1\/1 Running 0 8m1s<\/code><\/pre>\n\n\n\n In the direction of demonstrating the power of the cluster autoscaler and kube-downscaler combined, w’ll start by deploying and configuring the kube-downscaler operator in our cluster, in fact, this guy will shrink the application deployment immediately – as it\u2019s a non-working hour at the moment of writing this article \ud83d\ude09 –<\/p>\n\n\n\n
https:\/\/codeberg.org\/hjacobs\/kube-downscaler.git\ncd kube-downscaler<\/code><\/pre>\n\n\n\nThen, we will edit the configuration file: deploy\/deployment.yaml as the following: <\/p>\n\n\n\n
image: hjacobs\/kube-downscaler:20.10.0\n args:\n # dry run by default, remove to perform downscaling\n # - --dry-run\n # run every minute\n - --interval=30\n - --exclude-namespaces=kube-system\n - --exclude-deployments=kube-downscaler\n - --default-uptime=Mon-Fri 08:00-20:00 Europe\/Berlin\n - --include-resources=deployments,statefulsets,stacks,cronjobs\n - --deployment-time-annotation=deployment-time<\/code><\/pre>\n\n\n\nNow we rae ready to run our operator:<\/p>\n\n\n\n
kubectl apply -f deploy\/ <\/code><\/code><\/pre>\n\n\n\nAs expected, the Kube Downscaler scale down everything we deployed during the shift to 0 except the downscaler pod itself \ud83d\ude03<\/p>\n\n\n\n
|--\u2af8 kubectl get pods \n NAME READY STATUS RESTARTS AGE\n kube-downscaler-77cffd57fb-xjkpc 1\/1 Running 0 168m<\/code><\/pre>\n\n\n\nMoreover, we can see that there is only one node left in our cluster, thanks to the magic magic of the AKS Cluster autoscaler \ud83d\ude48<\/p>\n\n\n\n
|--\u2af8 kubectl get nodes \n NAME STATUS ROLES AGE VERSION\n aks-default-17028763-vmss000000 Ready agent 9d v1.19.3<\/code><\/pre>\n\n\n\nIn production, this tool helped us to scale down from ~13 to 3 worker nodes and this certainly makes a difference on the our Azure bill.<\/p>\n\n\n\n
Add a spot node pool to an AKS cluster<\/h3>\n\n\n\n
Next practice, Azure AKS costs can be reduced by using Spot instances as Kubernetes worker nodes. Spot Instances are available at up to a 90% discount compared to On-Demand prices.<\/p>\n\n\n\n
A spot node pool is a node pool backed by a spot virtual machine scale set<\/a>. Using spot VMs for nodes with your AKS cluster allows you to take advantage of unutilized capacity in Azure at a significant cost savings.<\/p>\n\n\n\nImportant 1: When deploying a spot node pool, Azure will allocate the spot nodes if there’s capacity available. But there’s no SLA for the spot nodes. A spot scale set that backs the spot node pool is deployed in a single fault domain and offers no high availability guarantees. At any time when Azure needs the capacity back, the Azure infrastructure will evict spot nodes.<\/p>\n\n\n\n
Important 2: Spot nodes are great for workloads that can handle interruptions, early terminations, or evictions. For example, workloads such as batch processing jobs, development and testing environments, and large compute workloads may be good candidates to be scheduled on a spot node pool.<\/p>\n\n\n\n
In this article, we will add a secondary spot node pool to an existing Azure Kubernetes Service (AKS) cluster.
Create a node pool using the az aks nodepool add<\/a>:
<\/p>\n\n\n\naz aks nodepool add \\\n--resource-group <yourResourceGroup> \\\n--cluster-name <yourAKSCluster> \\\n--name spotnodepool \\\n--priority Spot \\\n--eviction-policy Delete \\\n--spot-max-price -1 \\\n--min-count 1 \\\n--max-count 10 \\\n--no-wait<\/code><\/pre>\n\n\n\nBy default, you create a node pool with a priority<\/em> of Regular<\/em> in your AKS cluster when you create a cluster with multiple node pools. The above command adds an auxiliary node pool to an existing AKS cluster with a priority<\/em> of Spot<\/em>. The priority<\/em> of Spot<\/em> makes the node pool a spot node pool. The eviction-policy<\/em> parameter is set to Delete<\/em> in the above example, which is the default value. When you set the eviction policy<\/a> to Delete<\/em>, nodes in the underlying scale set of the node pool are deleted when they’re evicted. You can also set the eviction policy to Deallocate<\/em>. When you set the eviction policy to Deallocate<\/em>, nodes in the underlying scale set are set to the stopped-deallocated state upon eviction. Nodes in the stopped-deallocated state count against your compute quota and can cause issues with cluster scaling or upgrading. The priority<\/em> and eviction-policy<\/em> values can only be set during node pool creation. Those values can’t be updated later.<\/p>\n\n\n\nVerify the spot node pool<\/h4>\n\n\n\n
To verify your node pool has been added as a spot node pool:Azure CLICopy<\/p>\n\n\n\n
az aks nodepool show --resource-group <yourResourceGroup> --cluster-name <yourAKSCluster> --name spotnodepool\n<\/code><\/pre>\n\n\n\nConfirm scaleSetPriority<\/em> is Spot<\/em>.<\/p>\n\n\n\nTo schedule a pod to run on a spot node, add a toleration that corresponds to the taint applied to your spot node. The following example shows a portion of a yaml file that defines a toleration that corresponds to a kubernetes.azure.com\/scalesetpriority=spot:NoSchedule<\/em> taint used in the previous step.YAMLCopy<\/p>\n\n\n\nspec:\n containers:\n - name: spot-example\n tolerations:\n - key: \"kubernetes.azure.com\/scalesetpriority\"\n operator: \"Equal\"\n value: \"spot\"\n effect: \"NoSchedule\"\n ...\n<\/code><\/pre>\n\n\n\nWhen a pod with this toleration is deployed, Kubernetes can successfully schedule the pod on the nodes with the taint applied.<\/p>\n\n\n\n
Max price for a spot pool<\/h4>\n\n\n\n
Pricing for spot instances is variable<\/a>, based on region and SKU. For more information, see pricing for Linux<\/a> and Windows<\/a>.<\/p>\n\n\n\nWith variable pricing, you have option to set a max price, in US dollars (USD), using up to 5 decimal places. For example, the value 0.98765<\/em> would be a max price of $0.98765 USD per hour. If you set the max price to -1<\/em>, the instance won’t be evicted based on price. The price for the instance will be the current price for Spot or the price for a standard instance, whichever is less, as long as there is capacity and quota available.<\/p>\n\n\n\nLimitations<\/h4>\n\n\n\n
The following limitations apply when you create and manage AKS clusters with a spot node pool:<\/p>\n\n\n\n
\n- A spot node pool can’t be the cluster’s default node pool. A spot node pool can only be used for a secondary pool.<\/li>\n\n\n\n
- You can’t upgrade a spot node pool since spot node pools can’t guarantee cordon and drain. You must replace your existing spot node pool with a new one to do operations such as upgrading the Kubernetes version. To replace a spot node pool, create a new spot node pool with a different version of Kubernetes, wait until its status is\u00a0Ready<\/em>, then remove the old node pool.<\/li>\n\n\n\n
- The control plane and node pools cannot be upgraded at the same time. You must upgrade them separately or remove the spot node pool to upgrade the control plane and remaining node pools at the same time.<\/li>\n\n\n\n
- A spot node pool must use Virtual Machine Scale Sets.<\/li>\n\n\n\n
- You cannot change ScaleSetPriority or SpotMaxPrice after creation.<\/li>\n\n\n\n
- When setting SpotMaxPrice, the value must be -1 or a positive value with up to five decimal places.<\/li>\n\n\n\n
- A spot node pool will have the label\u00a0kubernetes.azure.com\/scalesetpriority:spot<\/em>, the taint\u00a0kubernetes.azure.com\/scalesetpriority=spot:NoSchedule<\/em>, and system pods will have anti-affinity.<\/li>\n\n\n\n
- You must add a\u00a0corresponding toleration<\/a>\u00a0to schedule workloads on a spot node pool.<\/li>\n<\/ul>\n\n\n\n
Clean up unused resources<\/h3>\n\n\n\n
Working in a fast-paced environment is great. We want tech organizations to accelerate. Faster software delivery also means more PR deployments, preview environments, prototypes, and spike solutions. All deployed on Kubernetes. Who has time to clean up test deployments manually? It’s easy to forget about deleting last week’s experiment. The cloud bill will eventually grow because of things we forget to shut down<\/p>\n\n\n\n
Kubernetes Janitor<\/a> (kube-janitor) helps to clean up your cluster. The janitor configuration is flexible for both global and local usage:<\/p>\n\n\n\n\n- Generic cluster-wide rules can dictate a maximum time-to-live (TTL) for PR\/test deployments.<\/li>\n\n\n\n
- Individual resources can be annotated with\u00a0janitor\/ttl, e.g. to automatically delete a spike\/prototype after 7 days.<\/li>\n<\/ul>\n\n\n\n
Generic rules are defined in a YAML file. Its path is passed via the –rules-file option to kube-janitor. Here an example rule to delete all namespaces with -pr- in their name after two days:<\/p>\n\n\n\n
- id: cleanup-resources-from-pull-requests \nresources:\n - namespaces \njmespath: \"contains(metadata.name, '-pr-')\" \nttl: 2d<\/code><\/pre>\n\n\n\nTo require the application label on Deployment and StatefulSet Pods for all new Deployments\/StatefulSet in 2020, but still allow running tests without this label for a week:<\/p>\n\n\n\n
- id: require-application-label \n# remove deployments and statefulsets without a label \"application\"<\/em> \nresources: \n- deployments - statefulsets # see http:\/\/jmespath.org\/specification.html<\/em> \njmespath: \"!(spec.template.metadata.labels.application) && metadata.creationTimestamp > '2020-01-01'\" \nttl: 7d<\/code><\/pre>\n\n\n\nTo run a time-limited demo for 30 minutes in a cluster where kube-janitor is running:<\/p>\n\n\n\n
kubectl run nginx-demo --image=nginx\nkubectl annotate deploy nginx-demo janitor\/ttl=30m\n<\/code><\/pre>\n\n\n\nAnother source of growing costs are persistent volumes (Azure ABS). Deleting a Kubernetes StatefulSet will not delete its persistent volumes (PVCs). Unused ABS volumes can easily cause costs of hundreds of dollars per month. Kubernetes Janitor has a feature to clean up unused PVCs. For example, this rule will delete all PVCs which are not mounted by a Pod and not referenced by a StatefulSet or CronJob:<\/p>\n\n\n\n
# delete all PVCs which are not mounted and not referenced by StatefulSets<\/em> \n- id: remove-unused-pvcs \nresources: \n- persistentvolumeclaims \njmespath: \"_context.pvc_is_not_mounted && _context.pvc_is_not_referenced\" \nttl: 24h \n<\/code><\/pre>\n\n\n\nKubernetes Janitor can help you keep your cluster “clean” and prevent slowly growing cloud costs. See the kube-janitor README fo<\/a>r instructions on how to deploy and configure.<\/p>\n\n\n\nOptimise your Kubernetes resource allocation<\/h3>\n\n\n\n
Kubernetes workloads specify their CPU\/memory needs via “resource requests”. CPU resources are measured in virtual cores or more commonly in “millicores”, e.g. 500m denoting 50% of a vCPU. Memory resources are measured in Bytes and the usual suffixes can be used, e.g. 500Mi denoting 500 Mebibyte. Resource requests “block” capacity on worker nodes, i.e. a Pod with 1000m CPU requests on a node with 4 vCPUs will leave only 3 vCPUs available for other Pods. <\/p>\n\n\n\n
Slack<\/em> is the difference between the requested resources and the real usage. For example, a Pod which requests 2 GiB of memory, but only uses 200 MiB, has ~1.8 GiB of memory “slack”. Slack costs money. We can roughly say that 1 GiB of memory slack costs ~$10\/month.<\/p>\n\n\n\nKubernetes Resource Report<\/a> (kube-resource-report) displays slack and can help you identify saving potential:<\/p>\n\n\n\n<\/figure>\n\n\n\nKubernetes Resource Report shows slack aggregated by application and team. This allows finding opportunities where resource requests can be lowered. The generated HTML report only provides a snapshot of resource usage. You need to look at CPU\/memory usage over time to set the right resource requests. Here a Grafana chart of a “typical” service with a lot of CPU slack: all Pods stay well below the 3 requested CPU cores:<\/p>\n\n\n\n<\/figure>\n\n\n\nReducing the CPU requests from 3000m to ~400m frees up resources for other workloads and allows the cluster to scale down.<\/p>\n\n\n\n
“The average CPU utilization of EC2 instances often hovers in the range of single-digit percentages” writes Corey Quinn<\/a>. While EC2 Right Sizing might be wrong<\/a>, changing some Kubernetes resource requests in a YAML file is easy to do and can yield huge savings.<\/p>\n\n\n\nBut do we really want humans to change numbers in YAML files? No, machines can do this better! The Kubernetes Vertical Pod Autoscaler<\/a> (VPA) does exactly this: adapt resource requests and limits to match the workload. Here some example graph of Prometheus’ CPU requests (thin blue line) adapted by VPA over time:<\/p>\n\n\n\n<\/figure>\n\n\n\nZalando uses the VPA in all its clusters<\/a> for infrastructure components. Non-critical applications can also use VPA.<\/p>\n\n\n\nFairwind’s Goldilocks<\/a> is a tool creating a VPA for each deployment in a namespace and then shows the VPA recommendation on its dashboard. This can help developers set the right CPU\/memory requests for their application(s):<\/p>\n\n\n\n<\/figure>\n\n\n\nUse horizontal autoscaling<\/h3>\n\n\n\n
Last, but not least, will introduce the HPA to save cloud costs with Kubernetes on Azure. Many applications\/services deal with a dynamic load pattern: sometimes their Pods idle and sometimes they are at their capacity. Running always with a fleet of Pods to handle the maximum peak load is not cost efficient. Kubernetes supports horizontal autoscaling via the HorizontalPodAutoscaler<\/a> (HPA) resource. CPU usage is often a good metric to scale on:<\/p>\n\n\n\napiVersion: autoscaling\/v2beta2\nkind: HorizontalPodAutoscaler\nmetadata:\n name: my-app\nspec:\n scaleTargetRef:\n apiVersion: apps\/v1\n kind: Deployment\n name: my-app\n minReplicas: 3\n maxReplicas: 10\n metrics:\n - type: Resource\n resource:\n name: cpu\n target:\n averageUtilization: 100\n type: Utilization\n <\/code><\/pre>\n\n\n\nConfiguring horizontal autoscaling with HPA should be one of the default actions to increase efficiency for stateless services. Spotify has a presentation with their learnings and recommendations for HPA: Scale Your Deployments, Not Your Wallet<\/a>.<\/p>\n\n\n\nThat\u2019s all folks!<\/h3>\n\n\n\n
That\u2019s all for this lab, thanks for reading \ud83d\ude4f
I hope you find some of the presented tools useful to reduce your cloud bill.<\/p>\n\n\n\n
References and credits<\/h3>\n\n\n\n\n- Featured image: Caylent<\/a><\/li>\n\n\n\n
- https:\/\/srcco.de\/posts\/saving-cloud-costs-kubernetes-aws.html<\/a><\/li>\n\n\n\n
- https:\/\/docs.microsoft.com\/en-us\/azure\/aks\/spot-node-pool<\/a><\/li>\n\n\n\n
- https:\/\/docs.microsoft.com\/en-us\/azure\/aks\/cluster-autoscaler<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"
This blog describe how you can save Cloud costs with Kubernetes on Azure while running at best performance and reliability.<\/p>\n","protected":false},"author":1,"featured_media":813,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[1],"tags":[],"class_list":["post-810","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-non-classe"],"yoast_head":"\n
Save cloud costs with Kubernetes on Azure<\/title>\n<meta name=\"description\" content=\"This blog describe how you can save Cloud costs with Kubernetes on Azure while running at best performance and reliability.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/aymen-segni.com\/index.php\/2020\/12\/23\/save-cloud-costs-with-kubernetes-on-azure\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Save cloud costs with Kubernetes on Azure\" \/>\n<meta property=\"og:description\" content=\"This blog describe how you can save Cloud costs with Kubernetes on Azure while running at best performance and reliability.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/aymen-segni.com\/index.php\/2020\/12\/23\/save-cloud-costs-with-kubernetes-on-azure\/\" \/>\n<meta property=\"og:site_name\" content=\"Run It On Cloud\" \/>\n<meta property=\"article:published_time\" content=\"2020-12-23T00:22:36+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-06-10T07:29:44+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/aymen-segni.com\/wp-content\/uploads\/2020\/12\/improve_cost.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"671\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"aymen-segni\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@https:\/\/x.com\/axsegni\" \/>\n<meta name=\"twitter:site\" content=\"@axsegni\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"aymen-segni\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"15 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/aymen-segni.com\/index.php\/2020\/12\/23\/save-cloud-costs-with-kubernetes-on-azure\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/aymen-segni.com\/index.php\/2020\/12\/23\/save-cloud-costs-with-kubernetes-on-azure\/\"},\"author\":{\"name\":\"aymen-segni\",\"@id\":\"https:\/\/aymen-segni.com\/#\/schema\/person\/32033966e7bd410bbaf1b79c7e94b59d\"},\"headline\":\"Save cloud costs with Kubernetes on Azure\",\"datePublished\":\"2020-12-23T00:22:36+00:00\",\"dateModified\":\"2024-06-10T07:29:44+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/aymen-segni.com\/index.php\/2020\/12\/23\/save-cloud-costs-with-kubernetes-on-azure\/\"},\"wordCount\":2557,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/aymen-segni.com\/#\/schema\/person\/32033966e7bd410bbaf1b79c7e94b59d\"},\"articleSection\":[\"Non class\u00e9\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/aymen-segni.com\/index.php\/2020\/12\/23\/save-cloud-costs-with-kubernetes-on-azure\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/aymen-segni.com\/index.php\/2020\/12\/23\/save-cloud-costs-with-kubernetes-on-azure\/\",\"url\":\"https:\/\/aymen-segni.com\/index.php\/2020\/12\/23\/save-cloud-costs-with-kubernetes-on-azure\/\",\"name\":\"Save cloud costs with Kubernetes on Azure\",\"isPartOf\":{\"@id\":\"https:\/\/aymen-segni.com\/#website\"},\"datePublished\":\"2020-12-23T00:22:36+00:00\",\"dateModified\":\"2024-06-10T07:29:44+00:00\",\"description\":\"This blog describe how you can save Cloud costs with Kubernetes on Azure while running at best performance and reliability.\",\"breadcrumb\":{\"@id\":\"https:\/\/aymen-segni.com\/index.php\/2020\/12\/23\/save-cloud-costs-with-kubernetes-on-azure\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/aymen-segni.com\/index.php\/2020\/12\/23\/save-cloud-costs-with-kubernetes-on-azure\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/aymen-segni.com\/index.php\/2020\/12\/23\/save-cloud-costs-with-kubernetes-on-azure\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Accueil\",\"item\":\"https:\/\/aymen-segni.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Save cloud costs with Kubernetes on Azure\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/aymen-segni.com\/#website\",\"url\":\"https:\/\/aymen-segni.com\/\",\"name\":\"Run It On Cloud\",\"description\":\"Accelerate your Cloud & MLOps Journey\",\"publisher\":{\"@id\":\"https:\/\/aymen-segni.com\/#\/schema\/person\/32033966e7bd410bbaf1b79c7e94b59d\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/aymen-segni.com\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":[\"Person\",\"Organization\"],\"@id\":\"https:\/\/aymen-segni.com\/#\/schema\/person\/32033966e7bd410bbaf1b79c7e94b59d\",\"name\":\"aymen-segni\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/aymen-segni.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/aymen-segni.com\/wp-content\/uploads\/2025\/02\/72799.jpg\",\"contentUrl\":\"https:\/\/aymen-segni.com\/wp-content\/uploads\/2025\/02\/72799.jpg\",\"width\":896,\"height\":1152,\"caption\":\"aymen-segni\"},\"logo\":{\"@id\":\"https:\/\/aymen-segni.com\/#\/schema\/person\/image\/\"},\"description\":\"Staff Engineer with over a decade of experience in building, scaling, and leading MLOPS, Cloud Native, SRE, and DevOps platforms across high-growth and enterprise environments. I specialize in architecting production-grade systems with a strong emphasis on resilience, security, and developer experience; bringing together deep expertise in distributed systems, Kubernetes, and modern platform engineering to empower engineering teams and accelerate business value. My work spans Cloud (AWS, GCP, Azure, OpenStack), Kubernetes, SRE (SLOs, observability, incident response), AI infrastructure and AgentOps (vLLM, Nvidia, RayServe, etc), and Platform Engineering (Backstage, Keptn, GitOps, self-service). I\u2019ve led teams through Cloud Native transformations, established scalable SRE practices, and built internal platforms that streamline operations and reduce cognitive load. With a strong programming background, and Infrastructure as Code (Terraform, Helm, Ansible), I drive automation-first approaches to eliminate toil, ensure reliability, and enable secure, compliant deployment pipelines. My focus today is on building Cloud Native AI platforms, where DevOps meets AI Infrastructure Stacks to support scalable, production-ready LLMs and AI Platforms. As a dedicated mentor, both within my teams and through platforms like MentorCruise, I am passionate about helping engineers perform at their best and assisting organizations in scaling with confidence. Driven by systems thinking, platform-as-a-product mindset, and engineering excellence, I help teams ship faster, operate smarter, and scale with confidence.\",\"sameAs\":[\"https:\/\/aymen-segni.com\",\"https:\/\/www.linkedin.com\/in\/aymen-segni\",\"https:\/\/twitter.com\/https:\/\/x.com\/axsegni\"],\"url\":\"https:\/\/aymen-segni.com\/index.php\/author\/admin8647\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Save cloud costs with Kubernetes on Azure","description":"This blog describe how you can save Cloud costs with Kubernetes on Azure while running at best performance and reliability.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/aymen-segni.com\/index.php\/2020\/12\/23\/save-cloud-costs-with-kubernetes-on-azure\/","og_locale":"en_US","og_type":"article","og_title":"Save cloud costs with Kubernetes on Azure","og_description":"This blog describe how you can save Cloud costs with Kubernetes on Azure while running at best performance and reliability.","og_url":"https:\/\/aymen-segni.com\/index.php\/2020\/12\/23\/save-cloud-costs-with-kubernetes-on-azure\/","og_site_name":"Run It On Cloud","article_published_time":"2020-12-23T00:22:36+00:00","article_modified_time":"2024-06-10T07:29:44+00:00","og_image":[{"width":1200,"height":671,"url":"https:\/\/aymen-segni.com\/wp-content\/uploads\/2020\/12\/improve_cost.png","type":"image\/png"}],"author":"aymen-segni","twitter_card":"summary_large_image","twitter_creator":"@https:\/\/x.com\/axsegni","twitter_site":"@axsegni","twitter_misc":{"Written by":"aymen-segni","Est. reading time":"15 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/aymen-segni.com\/index.php\/2020\/12\/23\/save-cloud-costs-with-kubernetes-on-azure\/#article","isPartOf":{"@id":"https:\/\/aymen-segni.com\/index.php\/2020\/12\/23\/save-cloud-costs-with-kubernetes-on-azure\/"},"author":{"name":"aymen-segni","@id":"https:\/\/aymen-segni.com\/#\/schema\/person\/32033966e7bd410bbaf1b79c7e94b59d"},"headline":"Save cloud costs with Kubernetes on Azure","datePublished":"2020-12-23T00:22:36+00:00","dateModified":"2024-06-10T07:29:44+00:00","mainEntityOfPage":{"@id":"https:\/\/aymen-segni.com\/index.php\/2020\/12\/23\/save-cloud-costs-with-kubernetes-on-azure\/"},"wordCount":2557,"commentCount":0,"publisher":{"@id":"https:\/\/aymen-segni.com\/#\/schema\/person\/32033966e7bd410bbaf1b79c7e94b59d"},"articleSection":["Non class\u00e9"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/aymen-segni.com\/index.php\/2020\/12\/23\/save-cloud-costs-with-kubernetes-on-azure\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/aymen-segni.com\/index.php\/2020\/12\/23\/save-cloud-costs-with-kubernetes-on-azure\/","url":"https:\/\/aymen-segni.com\/index.php\/2020\/12\/23\/save-cloud-costs-with-kubernetes-on-azure\/","name":"Save cloud costs with Kubernetes on Azure","isPartOf":{"@id":"https:\/\/aymen-segni.com\/#website"},"datePublished":"2020-12-23T00:22:36+00:00","dateModified":"2024-06-10T07:29:44+00:00","description":"This blog describe how you can save Cloud costs with Kubernetes on Azure while running at best performance and reliability.","breadcrumb":{"@id":"https:\/\/aymen-segni.com\/index.php\/2020\/12\/23\/save-cloud-costs-with-kubernetes-on-azure\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/aymen-segni.com\/index.php\/2020\/12\/23\/save-cloud-costs-with-kubernetes-on-azure\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/aymen-segni.com\/index.php\/2020\/12\/23\/save-cloud-costs-with-kubernetes-on-azure\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Accueil","item":"https:\/\/aymen-segni.com\/"},{"@type":"ListItem","position":2,"name":"Save cloud costs with Kubernetes on Azure"}]},{"@type":"WebSite","@id":"https:\/\/aymen-segni.com\/#website","url":"https:\/\/aymen-segni.com\/","name":"Run It On Cloud","description":"Accelerate your Cloud & MLOps Journey","publisher":{"@id":"https:\/\/aymen-segni.com\/#\/schema\/person\/32033966e7bd410bbaf1b79c7e94b59d"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/aymen-segni.com\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":["Person","Organization"],"@id":"https:\/\/aymen-segni.com\/#\/schema\/person\/32033966e7bd410bbaf1b79c7e94b59d","name":"aymen-segni","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/aymen-segni.com\/#\/schema\/person\/image\/","url":"https:\/\/aymen-segni.com\/wp-content\/uploads\/2025\/02\/72799.jpg","contentUrl":"https:\/\/aymen-segni.com\/wp-content\/uploads\/2025\/02\/72799.jpg","width":896,"height":1152,"caption":"aymen-segni"},"logo":{"@id":"https:\/\/aymen-segni.com\/#\/schema\/person\/image\/"},"description":"Staff Engineer with over a decade of experience in building, scaling, and leading MLOPS, Cloud Native, SRE, and DevOps platforms across high-growth and enterprise environments. I specialize in architecting production-grade systems with a strong emphasis on resilience, security, and developer experience; bringing together deep expertise in distributed systems, Kubernetes, and modern platform engineering to empower engineering teams and accelerate business value. My work spans Cloud (AWS, GCP, Azure, OpenStack), Kubernetes, SRE (SLOs, observability, incident response), AI infrastructure and AgentOps (vLLM, Nvidia, RayServe, etc), and Platform Engineering (Backstage, Keptn, GitOps, self-service). I\u2019ve led teams through Cloud Native transformations, established scalable SRE practices, and built internal platforms that streamline operations and reduce cognitive load. With a strong programming background, and Infrastructure as Code (Terraform, Helm, Ansible), I drive automation-first approaches to eliminate toil, ensure reliability, and enable secure, compliant deployment pipelines. My focus today is on building Cloud Native AI platforms, where DevOps meets AI Infrastructure Stacks to support scalable, production-ready LLMs and AI Platforms. As a dedicated mentor, both within my teams and through platforms like MentorCruise, I am passionate about helping engineers perform at their best and assisting organizations in scaling with confidence. Driven by systems thinking, platform-as-a-product mindset, and engineering excellence, I help teams ship faster, operate smarter, and scale with confidence.","sameAs":["https:\/\/aymen-segni.com","https:\/\/www.linkedin.com\/in\/aymen-segni","https:\/\/twitter.com\/https:\/\/x.com\/axsegni"],"url":"https:\/\/aymen-segni.com\/index.php\/author\/admin8647\/"}]}},"jetpack_featured_media_url":"https:\/\/aymen-segni.com\/wp-content\/uploads\/2020\/12\/improve_cost.png","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/aymen-segni.com\/index.php\/wp-json\/wp\/v2\/posts\/810","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/aymen-segni.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/aymen-segni.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/aymen-segni.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/aymen-segni.com\/index.php\/wp-json\/wp\/v2\/comments?post=810"}],"version-history":[{"count":41,"href":"https:\/\/aymen-segni.com\/index.php\/wp-json\/wp\/v2\/posts\/810\/revisions"}],"predecessor-version":[{"id":1089,"href":"https:\/\/aymen-segni.com\/index.php\/wp-json\/wp\/v2\/posts\/810\/revisions\/1089"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/aymen-segni.com\/index.php\/wp-json\/wp\/v2\/media\/813"}],"wp:attachment":[{"href":"https:\/\/aymen-segni.com\/index.php\/wp-json\/wp\/v2\/media?parent=810"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/aymen-segni.com\/index.php\/wp-json\/wp\/v2\/categories?post=810"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/aymen-segni.com\/index.php\/wp-json\/wp\/v2\/tags?post=810"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}