Affinity & Anti-Affinity Rules

You've built the cluster, deployed pods, and scaled them with ReplicaSets. But what happens when your database pod is scheduled on a node alongside your CPU-hungry video transcoder, or your required PCI-compliant pods end up on the same host as a noisy neighbor? Without explicit placement rules, the Kubernetes scheduler treats all nodes as equals — which can lead to performance bottlenecks, availability risks, or compliance violations. Affinity and anti-affinity rules give you surgical control over where your pods land, allowing you to co-locate or separate workloads based on node characteristics or other pod identities.

The problem this lesson solves

Default pod scheduling is distributed — the scheduler spreads pods across available nodes to balance resource usage. That's fine for stateless web apps, but many real-world scenarios demand targeted placement:

• Performance isolation: Keep your database pods away from batch processing pods that might eat CPU or memory.
• Availability & resilience: Spread replicas of a StatefulSet across different nodes, zones, or racks so a single failure doesn't take down the service.
• Compliance & data locality: Ensure pods that must stay in a specific region or on hardware with special security markings run only on those nodes.
• Cost optimization: Collocate pods that communicate heavily (e.g., app and cache) on the same node or in the same zone to reduce cross-node network traffic and latency.
• Hardware requirements: Some workloads need GPUs, fast SSDs, or large memory — you can pin them to nodes with those resources.

Pro Tip: Affinity and anti-affinity rules are declarative, not imperative. You express your intentions in the Pod spec, and the scheduler respects them during scheduling only. They don't prevent existing pods from being moved — for that, you need taints and tolerations.

Core concept / mental model

Think of the Kubernetes scheduler as a matchmaker. Without rules, it introduces pods to any node with enough resources. Affinity rules are like saying "I only want pods matched to nodes from this region" or "I want my frontend pods near my API pods." Anti-affinity rules say "Don't sit my database replicas on the same node."

There are two dimensions in Kubernetes:

• Node affinity: Constrains which nodes a pod can be scheduled on, based on node labels.
• Pod affinity / anti-affinity: Constrains scheduling based on the labels of other pods already running on the node.

Both support two enforcement levels: - requiredDuringSchedulingIgnoredDuringExecution — hard constraint: the pod will not be scheduled unless the condition is met. - preferredDuringSchedulingIgnoredDuringExecution — soft constraint: the scheduler tries to satisfy it, but will place the pod elsewhere if necessary.

The "IgnoredDuringExecution" part means that once a pod is running, changes to labels or node resources won't evict it. For runtime enforcement, you'd need a mutating admission webhook or custom controller.

How it works step by step

1. Label your nodes (or rely on existing labels)

Many cloud providers set labels like failure-domain.beta.kubernetes.io/zone, kubernetes.io/hostname, and beta.kubernetes.io/instance-type. You can also add custom labels:

kubectl label node worker-1 topology.kubernetes.io/zone=us-east-1a
kubectl label node worker-2 hw-type=gpu

2. Write the Pod spec with affinity rules

Node affinity example — run only on nodes with SSD storage:

apiVersion: v1
kind: Pod
metadata:
  name: fast-storage-pod
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: disk-type
            operator: In
            values:
            - ssd
  containers:
  - name: app
    image: nginx

Pod anti-affinity example — ensure replicas of a deployment aren't on the same node (high availability):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - web
            topologyKey: kubernetes.io/hostname
      containers:
      - name: nginx
        image: nginx

3. The scheduler matches labels with expressions

Inside matchExpressions, you use operators: - In / NotIn: set membership - Exists / DoesNotExist: presence of label key (value ignored) - Gt / Lt: greater-than or less-than (for numeric labels)

A pod with podAntiAffinity can use topologyKey — the label domain for placement. Common keys: - kubernetes.io/hostname — separate nodes - topology.kubernetes.io/zone — separate availability zones - topology.kubernetes.io/region — separate regions

Hands-on walkthrough

Let's deploy two microservices — api and cache — where: - api must run on GPU nodes (hw-type=gpu) - cache should be as far from other cache pods as possible (anti-affinity by zone)

Step 1: Label nodes

kubectl label node worker-2 hw-type=gpu
kubectl label node worker-3 hw-type=gpu

Step 2: Deploy API with node affinity

Create api-deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 2
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: hw-type
                operator: In
                values:
                - gpu
      containers:
      - name: api
        image: nginx

kubectl apply -f api-deployment.yaml

Verify pods land only on GPU nodes:

kubectl get pods -o wide | grep api

Step 3: Deploy cache with pod anti-affinity (by zone)

Assume your nodes have topology.kubernetes.io/zone labels (set by cloud provider or manually).

Create cache-deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: cache
spec:
  replicas: 3
  selector:
    matchLabels:
      app: cache
  template:
    metadata:
      labels:
        app: cache
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - cache
              topologyKey: topology.kubernetes.io/zone
      containers:
      - name: cache
        image: redis

kubectl apply -f cache-deployment.yaml
kubectl get pods -o wide | grep cache

You'll see each cache pod is scheduled to a different zone (if available). If only two zones exist, two pods land in separate zones and the third will share one — because it's preferred, not required.

Compare options / when to choose what

When to use each:

• Use node affinity when placement depends on node capabilities (GPU, SSD, region).
• Use pod affinity when communication latency or data locality matters (e.g., app + cache).
• Use pod anti-affinity for high availability, failure isolation, or noisy neighbor avoidance.

Note: requiredDuringScheduling rules can cause pods to remain Pending if no node matches. Always set preferred unless the workload absolutely cannot run elsewhere.

Troubleshooting & edge cases

• Pod stuck in Pending: Check kubectl describe pod <name> for events. Look for 0/X nodes are available: 1 node(s) didn't match node affinity. Ensure nodes have the required labels. If using requiredDuringScheduling, the pod won't schedule until a matching node exists.
• Wrong pods separated: Pod anti-affinity acts on pods with matching labels. If your cache and web pods share a label value (e.g., app: myapp), they'll anti-affect each other. Use unique label values per service.
• Too many rules slow scheduling: Complex matchExpressions with many values or nested conditions can make the scheduler iterate more. Keep expressions simple. For very large clusters, consider sticking to nodeSelector (which is simpler) when possible.
• Scheduling across zones fails: If your cluster doesn't have topology.kubernetes.io/zone labels on nodes (e.g., minikube or single-node cluster), anti-affinity by zone won't work. Add labels manually for testing, or use kubernetes.io/hostname for node-level separation.
• preferred rules ignored: A soft rule with weight: 1 is almost never considered. Use higher weight (e.g., 100) to make it more influential. Weights are summed across all soft rules; the scheduler scores nodes and picks the highest.

What you learned & what's next

You now understand how to use affinity and anti-affinity rules to control pod placement. You learned: - The difference between node affinity (pinning to hardware) and pod affinity/anti-affinity (co-location or separation with other pods). - Hard vs soft constraints and when to use each. - How to write YAML with matchExpressions and topologyKey. - Practical examples for high availability (pod anti-affinity) and hardware targeting (node affinity).

Next topic: Now that you can place pods precisely, you'll want to ensure they run with appropriate resources. Move to Resource limits and requests to guarantee your scheduled pods get the CPU and memory they need — without starving others.