Apply Taints and Tolerations to Nodes

You've built clusters, scheduled Pods, and maybe even tamed a DaemonSet. But what happens when a critical Pod must run on a specific node, or when a node is so special that nothing else should touch it? Without taints and tolerations, the scheduler treats every node as equally eligible—leading to resource conflicts, security leaks, or workloads landing on the wrong hardware. Let's fix that.

The problem this lesson solves

Imagine a cluster with a GPU-equipped node reserved exclusively for machine learning training. Without taints, the scheduler will scatter web servers and databases onto that expensive GPU machine, wasting capacity and blocking critical jobs. Or consider a node with specialized compliance software—allowing general Pods to land there violates audit requirements.

The default scheduling policy is permissive: if a node has resources, the scheduler will happily place Pods there. This becomes a liability when you need to dedicate nodes to specific workloads, restrict access to sensitive Pods, or protect critical infrastructure from accidental eviction. Taints and tolerations solve this by letting nodes repel Pods unless those Pods explicitly declare they can tolerate the node.

Core concept / mental model

Think of taints as "No Trespassing" signs on a node, and tolerations as the special badge that lets certain Pods pass through. A node can have one or more taints; a Pod must have matching tolerations to be scheduled on that node.

What is a taint?

A taint is a key-value pair with an effect attached to a node. The effect tells the scheduler what to do with Pods that don't tolerate the taint:

• NoSchedule — Don't schedule new Pods unless they tolerate it.
• PreferNoSchedule — Soft version: try to avoid, but not guaranteed.
• NoExecute — Evict existing Pods that don't tolerate, in addition to preventing new ones.

What is a toleration?

A toleration is a matching rule on a Pod that says "I can handle this taint." It must specify the same key, value (if the taint uses a value), and effect.

Pro tip: Tolerations are not permissions to choose a node—they only allow a Pod to be scheduled on a tainted node if it already matches other scheduling constraints. Use node affinity for positive selection, tolerations for negative filtering.

How it works step by step

1. Apply a taint to a node using kubectl taint nodes <node> key=value:effect.
2. When the scheduler evaluates node suitability for a Pod, it checks the node's taints.
3. If the Pod has a toleration matching all taints with effect NoSchedule or NoExecute, the node is considered schedulable.
4. For NoExecute, any running Pods that lack the toleration are evicted immediately.
5. PreferNoSchedule is advisory—the scheduler avoids the node but may still use it if no other node fits.

Taint structure

A taint is defined as key=value:Effect. The value is optional—you can have key:Effect (value empty). Multiple taints on a node are OR'ed: the Pod must tolerate all non-optional taints to be scheduled.

Toleration matching rules

A toleration can match by: - Exact key, value, and effect - Key and effect only (value omitted, signified by operator: "Exists") - Operator Exists with no key (tolerates everything, rarely used)

Hands-on walkthrough

Prerequisites

• A running Kubernetes cluster (minikube works)
• kubectl configured

Step 1: Taint a node

First, get your node names:

kubectl get nodes

Assume you have a node named gpu-node. Add a taint that says "only ML workloads allowed":

kubectl taint nodes gpu-node gpu=true:NoSchedule

Verify the taint:

kubectl describe node gpu-node | grep Taints

Expected output (truncated):

Taints:             gpu=true:NoSchedule

Step 2: Schedule a Pod without toleration

Create web-pod.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: web-server
spec:
  containers:
  - name: nginx
    image: nginx:1.25

Apply it:

kubectl apply -f web-pod.yaml
kubectl get pods -o wide

You'll see the Pod stuck in Pending because no node tolerates the taint (unless other untainted nodes exist). The gpu-node has NoSchedule, so the scheduler avoids it.

Step 3: Schedule a Pod with toleration

Create ml-pod.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: ml-training
spec:
  containers:
  - name: pytorch
    image: pytorch/pytorch:latest
    command: ["sleep", "3600"]
  tolerations:
  - key: "gpu"
    operator: "Equals"
    value: "true"
    effect: "NoSchedule"

Apply and check:

kubectl apply -f ml-pod.yaml
kubectl get pods -o wide

The Pod should land on gpu-node because its toleration matches the node taint.

Step 4: Test NoExecute

Add a NoExecute taint to a different node (e.g., db-node):

kubectl taint nodes db-node critical=true:NoExecute

Now any Pod running on db-node without a toleration critical=true:NoExecute will be evicted. Start a test Pod on db-node first (maybe using nodeName), then apply the taint—you'll see the Pod get terminated.

Compare options / when to choose what

When to use taints vs node affinity? - Taints + tolerations: Keep Pods off a node unless they specifically opt in. Good for specialized hardware, compliance zones. - Node affinity: Attract Pods to a node using labels. Good for ensuring Pods run on specific node groups like zone=us-east. - Combine both: Use node affinity to pull Pods to a node pool, and taints to prevent other Pods from landing there.

Troubleshooting & edge cases

Pod stuck in Pending

• Check node taints: kubectl describe node <node>
• Verify Pod tolerations match exactly (key, value, effect)
• Ensure operator is Equals (default) or Exists — misspelling Equal causes silent mismatch

Evicted pods after NoExecute

• Pods without matching toleration get deleted. Use kubectl get events --watch to see eviction
• Ensure critical system Pods (like kube-proxy, coredns) have tolerations for control-plane taints

Toleration doesn't match

• Taint value gpu=true:NoSchedule requires toleration with value: "true" and effect: "NoSchedule". Omitting value with operator: "Exists" works, but only if you set operator: "Exists" explicitly.
• Common mistake: using operator: Equal instead of operator: "Equals" (case-sensitive — Equal is invalid).

Node already tainted

• kubectl taint will append new taints unless you use --overwrite. To remove: kubectl taint nodes <node> key:NoSchedule- (hyphen at the end).

What you learned & what's next

You now understand the core purpose of taints and tolerations: giving nodes the ability to repel Pods and letting Pods opt in to restricted nodes. You've applied NoSchedule, PreferNoSchedule, and NoExecute taints, scheduled both non-tolerant and tolerant Pods, and seen eviction in action. You can now implement dedicated node pools, protect sensitive hardware, and enforce scheduling policies.

Next up: Master node affinity to actively pull Pods toward specific nodes using labels — the counterpart to taints. In the next lesson, you'll combine both concepts to build precise, production-grade scheduling rules.