DaemonSet on Every Node

You’ve deployed Deployments that keep a desired number of Pods alive, and StatefulSets that manage sticky identity for stateful workloads. But what happens when you need every node in your cluster to run a specific Pod — like a logging agent, a monitoring collector, or a security scanner? A normal Deployment won't guarantee a Pod lands on each node, and you'd have to manually scale or use complex affinity rules. That's exactly where a DaemonSet shines: it ensures one copy of a Pod runs on every node (or a subset, if you use node selectors). Without it, you'd be constantly rebalancing workloads or missing critical coverage on new nodes.

The problem this lesson solves

Imagine you run a 50-node Kubernetes cluster. You need a log-forwarder (like Fluentd or Filebeat) on every machine to ship logs to a central store. With a Deployment, you could run 50 replicas, but if a node is added or removed, you'd need to manually adjust the replica count. Worse, during rolling updates, Pods might land on the same node, leaving others uncovered. A DaemonSet automates this: whenever a new node joins the cluster, a new Pod is created on it; when a node is removed, the Pod is garbage-collected. The core pain is ensuring per-node infrastructure runs reliably without manual intervention.

Core concept / mental model

Think of a DaemonSet as a "one-per-node" guarantee. Whereas a Deployment ensures a total replica count across the cluster, a DaemonSet ties the number of replicas to the number of nodes. The controller watches the node list and creates or deletes Pods to match. Key properties:

• Node-level services: DaemonSets are ideal for workloads that must run on every node (or a targeted subset), such as log collectors, node monitoring daemons, CNI plugins, or storage daemons.
• Automatic scaling: If you scale your cluster up or down, the DaemonSet adjusts without any action from you.
• Scheduling control: DaemonSets respect node selectors, taints/tolerations, and affinity rules — you can limit which nodes run the Pod.

Pro tip: A DaemonSet’s Pod is created on a node before the node is marked Ready in some cases (e.g., for CNI plugins). This makes DaemonSets essential for cluster bootstrapping.

How it works step by step

1. You define a DaemonSet YAML with a Pod template (similar to a Deployment, but without replicas).
2. Kubernetes DaemonSet controller watches the node API — it maintains a list of all nodes that match the DaemonSet’s node selector (default: all nodes).
3. For each matching node, the controller ensures exactly one Pod is running. If no Pod exists, it creates one; if the Pod dies, it re-creates it.
4. When a node is removed, the Pod is automatically deleted (no manual cleanup).
5. If a DaemonSet updates its Pod template, the controller rolls out new Pods node-by-node (configurable with updateStrategy).

Key differences from a Deployment

Hands-on walkthrough

Let's create a simple DaemonSet that runs an Nginx Pod on every node. We'll verify it works and then clean up.

1. Create a DaemonSet YAML

Save this as nginx-daemonset.yaml:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nginx-per-node
  labels:
    app: nginx
spec:
  selector:
    matchLabels:
      name: nginx
  template:
    metadata:
      labels:
        name: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.21
        ports:
        - containerPort: 80

Note: No replicas field — the controller handles that.

2. Apply the DaemonSet

kubectl apply -f nginx-daemonset.yaml

3. Verify Pods run on every node

kubectl get daemonset
# Output (example for 3-node cluster):
# NAME              DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
# nginx-per-node    3         3         3       3            3           <none>          10s

Check each Pod's node:

kubectl get pods -o wide | grep nginx
# Output:
# nginx-per-node-abc12   node-1
# nginx-per-node-def34   node-2
# nginx-per-node-ghi56   node-3

You should see one Pod per node. If you have a single-node cluster (e.g., Minikube), you'll see exactly one Pod.

4. Clean up

kubectl delete daemonset nginx-per-node

Compare options / when to choose what

When to choose DaemonSet: - You need a Pod on every node (e.g., node-exporter, kube-proxy). - You want automatic scaling with cluster size. - You need simple rolling updates per node.

When not to: - You need exactly N Pods total (use Deployment). - You need ordering between Pods (use StatefulSet). - You are managing control-plane components (use static Pods or systemd).

Troubleshooting & edge cases

Pod not created on a node

Check if the node has taints that your DaemonSet Pod doesn't tolerate. By default, DaemonSet Pods ignore node.kubernetes.io/unschedulable but respect other taints (e.g., node-role.kubernetes.io/control-plane). To run on master nodes, add tolerations:

tolerations:
- key: node-role.kubernetes.io/control-plane
  operator: Exists
  effect: NoSchedule

DaemonSet starts Pod before node is Ready

This is by design — necessary for CNI and critical add-ons. If your Pod requires something that isn't available, set node.kubernetes.io/not-ready toleration.

Rolling update hangs

If a Pod fails to start on a node, the DaemonSet controller stops the rollout until the Pod becomes Ready. Check Pod logs with kubectl logs <pod-name>. Common issues: image pull errors, resource limits, or volume mounts.

One node runs two Pods

If you see more than one Pod per node, check that your selector.matchLabels is unique. Multiple DaemonSets can select overlapping labels, but each DaemonSet ensures exactly one per node. Use unique labels per DaemonSet.

What you learned & what's next

You now understand how to use a DaemonSet to run a Pod on every node — from the core concept to a hands-on YAML deployment. You can troubleshoot common issues like taints and update strategies. This knowledge is crucial for deploying cluster-level services like log collectors, monitoring agents, and networking components.

Next lesson: Learn about Jobs and CronJobs for running batch workloads to completion. With DaemonSets under your belt, you're ready to tackle ephemeral tasks that need to run once or on a schedule.