Enable cluster autoscaling

You’ve built a deployment, exposed it with a Service, and maybe even rolled out a new version. But what happens when traffic spikes? Your pods get scaled up by the Horizontal Pod Autoscaler, but if every node in your cluster is already full, those pods will stay Pending indefinitely — stuck waiting for resources that don’t exist. That’s the exact pain cluster autoscaling solves: it automatically adds or removes nodes so your cluster always has the right amount of compute capacity for your workloads. Without it, you’re either overprovisioning (wasting money) or underprovisioning (breaking SLOs). This lesson shows you how to enable and configure cluster autoscaling on a real Kubernetes cluster.

The problem this lesson solves

Imagine a Black Friday sale. Your web app usually runs on 3 nodes, each with 4 CPU cores and 8 GB of RAM. Suddenly, user traffic triples. The HPA kicks in and schedules 15 more pods — but the cluster scheduler finds zero nodes with enough free CPU or memory. Those 15 pods enter Pending state, the autoscaler keeps them there because there’s no room, and your users see a spinning loader or an error page.

Manually adding nodes is slow and error-prone. You can’t sit at a terminal 24/7. Cloud providers charge per second, so leaving idle nodes running costs real money. Cluster autoscaling treats node capacity like a dynamic pool: shrink when demand drops, grow when demand spikes — all without human intervention.

Key problems it solves: - Pending pods: scale out nodes when the scheduler can’t place a pod. - Idle capacity waste: scale in nodes when they have no workloads (respecting disruption budgets). - Infrastructure cost control: only pay for nodes you need, right now.

Core concept / mental model

Think of a Kubernetes cluster like a parking garage. Each node is a parking spot; each pod is a car. If all spots are full, new cars queue at the entrance. Cluster autoscaling is the garage manager who sees the queue getting long and builds a new parking level — or, when cars leave, demolishes empty levels.

Technically, cluster autoscaling is a control loop that:

1. Watches for Pending pods (pods the scheduler couldn’t place because of insufficient resources).
2. Asks the cloud provider (AWS, GCP, Azure, etc.) for a new node matching the pending pod’s requirements.
3. Adds the node to the cluster, then re-schedules the pending pods.
4. Periodically checks for underutilized nodes (e.g., CPU utilization < 50% for 10+ minutes) and safely drains & removes them.

Pro tip: Cluster autoscaling is NOT the same as the Horizontal Pod Autoscaler. HPA scales pods; cluster-autoscaler scales nodes. They complement each other. HPA + cluster autoscaler = fully elastic workloads.

How it works step by step

Step 1 — Install the cluster-autoscaler

The official deployment lives in the autoscaler GitHub repo. You install it as a Deployment in the kube-system namespace.

# Example for AWS (EKS) — adjust for your cloud
deploy/kubernetes/cluster-autoscaler-autodiscover.sh

Step 2 — Configure autoscaling groups

Cluster autoscaler works with node groups — a logical set of nodes (e.g., all nodes of the same instance type in an ASG on AWS, managed instance group on GCP, VMSS on Azure).

Key flags passed to the autoscaler binary: - --nodes=0:100:nodes-base — min:max:node-group-name - --scale-down-delay-after-add=10m — wait 10 minutes after a new node before considering scale-down - --scale-down-unneeded-time=5m — if a node is underutilized for 5 minutes, consider removing it

Step 3 — Test with a resource hunger pod

Create a Deployment that requests more memory than any single node can provide:

# big-pod.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: memory-hog
spec:
  replicas: 1
  selector:
    matchLabels:
      app: memory-hog
  template:
    metadata:
      labels:
        app: memory-hog
    spec:
      containers:
      - name: hog
        image: polinux/stress
        resources:
          requests:
            memory: "100Gi"   # bigger than any single node's capacity

Apply it and watch the magic:

kubectl apply -f big-pod.yaml
# Pod will be Pending because scheduler can't find a node with 100Gi
kubectl get events -w
# A new node will be provisioned automatically

After a few minutes, you’ll see a new node join:

kubectl get nodes
# New node appears (e.g., ip-10-0-1-200.ec2.internal   Ready)

Step 4 — Scale down

Delete the Deployment and wait:

kubectl delete deployment memory-hog
# Wait ~10 minutes for scale-down delay
# Node will be cordoned, drained, then terminated

Hands-on walkthrough

Let’s enable cluster autoscaling on a real cluster. We’ll use minikube with the gce driver (simulates cloud) — but the steps work identically on EKS, GKE, AKS.

Prerequisites

• A running Kubernetes cluster (minikube, EKS, GKE, AKS)
• kubectl configured
• Cloud provider credentials (if using managed k8s)

1 — Check current nodes

kubectl get nodes --label-columns=node.kubernetes.io/instance-type
# List of nodes and their types

2 — Deploy cluster-autoscaler

On GKE (simplest example):

kubectl create deployment cluster-autoscaler \
  --image=registry.k8s.io/autoscaling/cluster-autoscaler:v1.29.0 \
  --namespace=kube-system

For the full YAML, use the official deployment manifest:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/autoscaler/master/cluster-autoscaler/cloudprovider/aws/examples/cluster-autoscaler-autodiscover.yaml

3 — Verify the pod is running

kubectl -n kube-system get pods -l app=cluster-autoscaler

4 — Simulate node pressure

Create a Deployment that requests 12 CPU across multiple pods:

# cpu-stress.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cpu-stress
spec:
  replicas: 3
  selector:
    matchLabels:
      app: cpu-stress
  template:
    metadata:
      labels:
        app: cpu-stress
    spec:
      containers:
      - name: stress
        image: polinux/stress
        command: ["stress"]
        args: ["--cpu", "4"]
        resources:
          requests:
            cpu: "4"
          limits:
            cpu: "4"

Apply:

kubectl apply -f cpu-stress.yaml
kubectl get pods -w
# Watch them go from Pending → Running as new nodes appear

5 — Watch cluster-autoscaler logs

kubectl -n kube-system logs -l app=cluster-autoscaler --tail=50
# Look for lines like:
# "Scale down: removing empty node " or "Scale up: adding 1 node(s)"

Compare options / when to choose what

When to use cluster autoscaler: - You have HPA-enabled workloads that can drive high resource usage - You want to avoid paying for idle nodes during off-peak - You need consistent node types and lifetimes

When to avoid it: - Single-node clusters (it will scale from 1 to 0 and your workloads die) - Stateful applications with local storage (nodes are ephemeral; use PersistentVolumes)

Troubleshooting & edge cases

“Pending pods but cluster autoscaler doesn’t scale up”

• Check the autoscaler logs: kubectl -n kube-system logs <pod-name> — look for no node group found or insufficient permissions
• Ensure the node group has minimum size > 0 and maximum size > current

“Node is scaled down but pods were evicted”

• Cluster autoscaler follows PodDisruptionBudgets — if you don’t set one, it may drain pods that can’t be safely moved. Always set PDB for workloads:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: web

“Scale down never happens”

• Check --scale-down-unneeded-time and --scale-down-delay-after-add flags — with default 10m, you must wait that long after no new pods appear.
• A node is “unneeded” only if all pods hosted can run on other nodes without overcommit.

“Autoscaler doesn’t see pending pods”

• Ensure the pod’s resource requests are set. If you only set limits, the scheduler may still place the pod on a node with insufficient capacity.
• Cluster autoscaler only reacts to pending pods — not to high CPU/memory utilization. If nodes are 90% full but pods can still be scheduled, it won’t scale up.

What you learned & what's next

You now understand how to enable cluster autoscaling — from the mental model of dynamic node pools to hands-on configuration and troubleshooting. Key takeaways:

• Cluster autoscaling adds nodes when pods are pending and removes them when underutilized.
• It works with HPA but on a different axis (nodes vs. pods).
• Configuration requires cloud provider permissions and proper node group setup.
• Always test with a workload that exceeds current capacity.
• Monitor logs and set up alerts for scaling events.

In the next lesson, you’ll learn how to apply resource quotas and limits at the namespace level — a crucial companion to autoscaling that prevents noisy neighbors from consuming all cluster resources.