Kubernetes Nodes and Clusters

You've probably heard that Kubernetes is the "operating system of the cloud," but understanding how its building blocks—nodes and clusters—actually work is the difference between heroically copy-pasting YAML and confidently deploying production workloads. Without a solid mental model, you'll struggle with debugging failed pods, scaling applications, or even understanding why your kubectl commands sometimes fail. Let's fix that right now.

The problem this lesson solves

Many developers start Kubernetes by learning high-level objects like Deployments, Services, or Ingresses. While those are essential, the underlying node and cluster architecture is what determines whether those objects run correctly. When a Pod fails to start because of resource constraints, or a Node goes offline, or your cluster's control plane becomes unresponsive—if you don't understand nodes and clusters, you're debugging blind. This lesson strips away the abstraction to give you a clear mental model of the physical (or virtual) machines that run your containers.

Core concept / mental model

Think of a Kubernetes cluster as a tiny data center. It has two kinds of machines:

• Master nodes (control plane nodes): The brains of the operation. They manage the cluster state, schedule workloads, respond to API requests, and ensure the desired state matches the actual state. They run critical components like kube-apiserver, kube-controller-manager, and etcd.
• Worker nodes: The muscle. They run your application containers. Each worker node runs kubelet (the node agent), kube-proxy (network rules), and a container runtime like containerd or CRI-O.

🎯 Pro Tip: A "node" in Kubernetes is not a server or a VM—it's the unit of compute capacity registered with the cluster. At the infrastructure level, a node might be a bare-metal server, a VM, or even a Raspberry Pi.

Key Definitions

• Node: A worker machine (or VM) that runs your Pods. Every node has a unique name (often its hostname) and a status—Ready, NotReady, or Unknown.
• Cluster: A set of nodes joined together, managed by a single control plane. The cluster is the boundary for networking, scheduling, and resource management.
• Control Plane: The collection of processes that manage the cluster. It's sometimes called the "master." In a production setup, you'd have multiple control plane nodes for high availability.

Diagram in Words

Imagine you have 3 nodes: one master (control plane) and two workers. The master node runs the API server, scheduler, and etcd. The two worker nodes each run Kubelet and a container runtime. When you apply a Deployment YAML, the API server stores the desired state in etcd, the scheduler picks a worker node with available CPU/memory, and the Kubelet on that node instructs the container runtime to start the Pods. The cluster is the entire system—master + all workers.

How it works step by step

1. Cluster Initialization: You use kubeadm init or a cloud provider (like eksctl or gcloud container clusters create) to create a control plane. This sets up kube-apiserver, etcd, and other components on one node.
2. Node Registration: Worker nodes run kubeadm join (or similar) to register themselves with the control plane. The API server receives a certificate signing request, validates it, and adds the node to the cluster's internal registry.
3. Pod Scheduling: When you create a Pod (or a Deployment that creates Pods), the scheduler filters nodes that meet requirements (CPU/memory, taints/tolerations, node selectors). Then it scores the remaining nodes and picks one. It updates the Pod's nodeName field.
4. Workload Execution: The Kubelet on the chosen node detects the new Pod binding. It uses the container runtime to pull images and start containers. The kube-proxy updates iptables or IPVS rules so the Pod can be reached via Service IPs.
5. Health Monitoring: Kubelet reports node status periodically (default 10-second heartbeat). If a node fails to report, the control plane marks it NotReady after a configurable timeout (default 40 seconds). Pods on a dead node will be rescheduled after a tolerationSeconds period.

Component Interaction Table

Hands-on walkthrough

Let's explore your own cluster (or use minikube if you don't have one). We'll inspect nodes, check their role, and see how they work.

Example 1: List all nodes in your cluster

kubectl get nodes -o wide

Expected output:

NAME               STATUS   ROLES           AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
minikube           Ready    control-plane   10d   v1.28.3   192.168.49.2   <none>        Ubuntu 22.04.3 LTS   5.15.0-91-generic   docker://24.0.7

Notice: - The node has ROLES=control-plane—it's a single-node cluster where the control plane and worker functions are combined. - INTERNAL-IP is the node's internal address (used for Pod-to-Node communication). - CONTAINER-RUNTIME shows docker://24.0.7 (though newer clusters use containerd://).

Example 2: Describe a node to see its details

kubectl describe node minikube | head -50

Partial output:

Name:               minikube
Roles:              control-plane
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/hostname=minikube
                    node-role.kubernetes.io/control-plane=
Annotations:        kubeadm.alpha.kubernetes.io/cri-socket: unix:///var/run/cri-dockerd.sock
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Fri, 01 Dec 2024 10:00:00 +0000
Taints:             <none>
Unschedulable:      false
Capacity:
  cpu:                4
  memory:             3930720Ki
  pods:               110
Allocatable:
  cpu:                4
  memory:             3868520Ki
  pods:               110

Key takeaways: - Capacity = raw hardware resources (CPU, memory, max Pods). - Allocatable = capacity minus resources reserved for system daemons and Kubelet overhead. - Taints (here <none>) prevent Pods from being scheduled unless they tolerate the taint. - Unschedulable set to false means the node can receive new Pods.

Example 3: Check node conditions

kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}'

Output:

minikube    True

The Ready condition must be True for the node to accept Pods.

Example 4: Simulate a node becoming unhealthy

Exercise: If you have a multi-node cluster (or use kind), try draining a node:

kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data
kubectl get nodes

You'll see the node become SchedulingDisabled and its status change to Ready,SchedulingDisabled. Then uncordon it to re-enable scheduling:

kubectl uncordon <node-name>

Compare options / when to choose what

When to choose what: - Always prefer multiple control plane nodes for anything beyond development—especially if you have many worker nodes or critical workloads. - For bare-metal or edge, consider using lightweight nodes with kubelet and spot instances. - If your cluster spans multiple availability zones, ensure data-plane nodes are evenly distributed.

Troubleshooting & edge cases

1. Node not joining the cluster

• Symptom: kubeadm join fails with "TLS handshake timeout."
• Fix: Ensure port 6443 is open on the control plane node. Run: bash netstat -tulpn | grep 6443 If not listening, check if kube-apiserver crashed. Also verify the token hasn't expired (kubeadm token create --ttl 2h).

2. Node shows NotReady

• Symptom: Node status is NotReady with conditions like NetworkUnavailable or KubeletHasDiskPressure.
• Common causes:
• Network plugin missing: If you didn't install a CNI (like Calico, Flannel, Weave), Pods can't communicate and the control plane marks the node Ready only after network is healthy.
• Disk pressure: Node runs out of disk space (or inodes). Run df -h and df -i and free up space.
• PID pressure: Too many processes. Increase kernel.pid_max or kill unwanted processes.

kubectl describe node <node-name> | grep -A5 Conditions

3. Pods stuck in Pending

• Symptom: Pod never gets scheduled.
• Check: Do any nodes have enough allocatable resources? bash kubectl describe node | grep -E "(Allocated|Allocatable)" kubectl describe pod <pod-name> | grep -A10 Events
• Fix: Remove taints, increase node capacity, or adjust resource requests in the Pod spec.

4. Control plane node fails

• Symptom: kubectl commands time out or return connection refused.
• Single control plane: The cluster is effectively down. To recover, you must restart the API server or reconfigure high availability.
• Multiple control planes: The other control plane nodes continue serving requests; the failed node is removed from etcd quorum. You can rebuild the failed node and rejoin it.

What you learned & what's next

You now understand: - The difference between control plane nodes and worker nodes. - The components that run on each node (kube-apiserver, etcd, kubelet, kube-proxy, container runtime). - How node registration and Pod scheduling work step by step. - How to inspect nodes using kubectl get nodes, kubectl describe node, and jsonpath. - How to troubleshoot common node and cluster issues.

This knowledge is the foundation for everything else: Deployments, Services, Ingress, and especially debugging when things go wrong. In the next lesson, you'll dive into node maintenance—how to safely drain, cordon, and upgrade nodes, and how to handle node failures gracefully.