StatefulSets for Python Services

Your Python service needs a stable identity. Maybe it's a worker that must always connect to the same database shard, or a leader-election participant where pods must know their exact names. A standard Deployment gives you ephemeral pods with random names and shared storage — a nightmare for stateful workloads. Kubernetes has a dedicated controller for this: StatefulSets. In this lesson, you'll learn how to deploy Python services that require stable network identities, persistent storage, and ordered operations — and when to reach for StatefulSets instead of Deployments.

The Problem This Lesson Solves

Imagine you run a Python-based distributed cache using Redis-like logic (or the actual Redis). Each pod must store data locally and, when it restarts, it must attach to the same storage volume and keep the same DNS name so clients don't break. With a Deployment:

• Pods get random suffixes like cache-5f4d6c7c9-8j2k3 — impossible to address individually.
• Each replica gets a new PersistentVolumeClaim (PVC) only if you manually attach it — but Deployments don't guarantee stable storage per replica.
• Scaling up or down doesn't ensure any ordering — pods can be deleted or created in any sequence, breaking leader-election or cluster membership protocols.

Typical Python services affected: databases (e.g., PostgreSQL, MySQL), message brokers (e.g., RabbitMQ, Kafka), distributed caches (Redis), and any service using quorum-based consistency (like etcd). But also your own stateful Python apps — for example, a data ingestion service that must track its own progress file that survives restarts.

The solution is a StatefulSet — a Kubernetes controller designed specifically for stateful applications.

Core Concept / Mental Model

Think of a StatefulSet as a litter of puppies — each pup has a distinct name and its own food bowl (persistent storage). Unlike Deployment's generic "worker bees" with identical names, StatefulSet pods have:

• Stable, unique network identities — each pod gets a predictable name: myapp-0, myapp-1, myapp-2. The name persists across rescheduling.
• Stable, persistent storage — each pod gets its own PersistentVolumeClaim, so data survives restarts. If a pod dies, the newly created pod reattaches to the same PVC.
• Ordered, graceful deployment and scaling — pods are created and terminated in sequentional order (first -0, then -1). This is crucial for applications that need initial setup before subsequent replicas start (e.g., a primary node before replicas).

Key Terminology

• StatefulSet Controller — watches the desired state and creates/recreates pods in order.
• Headless Service — a Service without a cluster IP that gives each pod a stable DNS entry (e.g., podname.headless-svc.namespace.svc.cluster.local).
• PersistentVolumeClaim (PVC) — per-pod claim that binds to a PersistentVolume (PV), either dynamically provisioned or pre-created.
• Pod Management Policy — either OrderedReady (default) or Parallel — controls whether pods are created/terminated in sequence.

How It Works Step by Step

1. Define a Headless Service — StatefulSets rely on a headless service (clusterIP: None) to give each pod a stable DNS name. Without it, clients can't reach individual pods by name.
2. Create a StatefulSet manifest — specify serviceName, replicas, a pod template, and a volumeClaimTemplates section that defines the PVC template.
3. Kubernetes creates pods sequentially — by default (OrderedReady), pod myapp-0 is created and must become ready before myapp-1 starts. This ordering is essential for applications like databases where the first node must initialize the cluster.
4. Each pod gets its own PVC — the controller creates a PVC from the template with a name like data-myapp-0, data-myapp-1, etc. If a pod is deleted, its replacement gets the same PVC.
5. Scaling and deletion — scaling down removes pods in reverse order (myapp-2 first, then myapp-1). No automatic cleanup of Storage classes, so volume claims may remain.

Pro tip: Always provide a podManagementPolicy if you don't need ordered startup — set it to Parallel to speed up deployments for stateless-like stateful apps (e.g., multiple cache nodes).

Hands-On Walkthrough

Let's deploy a simple Python service that writes to a local file (simulating state). We'll use a StatefulSet to ensure each replica has its own persistent volume.

1. Create the Python app

Create a small Flask app that increments a counter and writes it to a mounted volume.

# app.py
from flask import Flask
import os

app = Flask(__name__)
COUNTER_FILE = "/data/counter.txt"

@app.route("/")
def read_counter():
    if os.path.exists(COUNTER_FILE):
        with open(COUNTER_FILE, "r") as f:
            count = int(f.read().strip() or "0")
    else:
        count = 0
    return f"Pod {os.environ.get('POD_NAME', 'unknown')} counter: {count}\n"

@app.route("/increment")
def increment():
    if os.path.exists(COUNTER_FILE):
        with open(COUNTER_FILE, "r") as f:
            count = int(f.read().strip() or "0")
    else:
        count = 0
    count += 1
    with open(COUNTER_FILE, "w") as f:
        f.write(str(count))
    return f"Counter incremented to {count}\n"

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8080)

2. Create the Dockerfile

FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app.py .
CMD ["python", "app.py"]

3. Push to a registry (assume you have one)

docker build -t youruser/counter-app:v1 .
docker push youruser/counter-app:v1

4. Create a Headless Service & StatefulSet manifest

# statefulset.yaml
apiVersion: v1
kind: Service
metadata:
  name: counter-hsvc
spec:
  clusterIP: None
  selector:
    app: counter
  ports:
    - port: 8080
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: counter
spec:
  serviceName: counter-hsvc
  replicas: 3
  selector:
    matchLabels:
      app: counter
  template:
    metadata:
      labels:
        app: counter
    spec:
      containers:
        - name: counter
          image: youruser/counter-app:v1
          ports:
            - containerPort: 8080
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
          volumeMounts:
            - name: data
              mountPath: /data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 1Gi

5. Deploy and test

kubectl apply -f statefulset.yaml
kubectl get statefulset
kubectl get pods

Expected output:

NAME      READY   STATUS    RESTARTS   AGE
counter-0   1/1     Running   0          10s
counter-1   1/1     Running   0          5s
counter-2   1/1     Running   0          2s

Test that each pod has its own counter:

kubectl exec counter-0 -- curl localhost:8080/increment
kubectl exec counter-0 -- curl localhost:8080/increment
kubectl exec counter-1 -- curl localhost:8080/increment

You'll see counter-0 increments independently from counter-1. Also verify stable DNS names:

kubectl exec counter-0 -- nslookup counter-0.counter-hsvc.default.svc.cluster.local

Compare Options / When to Choose What

Variations

• Operators — For complex stateful apps (e.g., Kafka, PostgreSQL), consider community operators like Strimzi or Zalando's Postgres Operator. They handle backups, scaling, and resilience automatically.
• Headless service with publishNotReadyAddresses — if you need pods to be resolvable even when not ready, set this to true.
• StorageClass configuration — define a specific storageClassName in volumeClaimTemplates to control volume types (e.g., SSD vs HDD).

Troubleshooting & Edge Cases

• PVC stuck in Pending — often due to no StorageClass provisioner. Check with kubectl get pvc and kubectl describe pvc. Ensure a default StorageClass exists or specify storageClassName in your template.
• Pods stuck in ContainerCreating — volume mount errors (e.g., wrong path) or permission issues. Check kubectl describe pod for events.
• Ordered startup never completes — if pod -0 never becomes ready, -1 won't start. Check readiness probes and application health. For rapid testing, set podManagementPolicy: Parallel.
• Stale PVCs after scale-down — StatefulSets don't delete PVCs automatically. If you scale down to 0, your data remains; you must manually delete PVCs if you want cleanup.
• DNS resolution fails — ensure the headless service exists and selector matches pod labels. Test with kubectl exec and nslookup.

What You Learned & What's Next

You now understand why StatefulSets for Python services are essential for workloads that require stable identities and persistent storage. You learned to:

• Explain the core idea behind StatefulSets: stable network identities, per-pod PVCs, and ordered operations.
• Deploy a Python service as a StatefulSet, including the headless service and volumeClaimTemplates.
• Compare StatefulSets and Deployments to choose the right controller for your Python app.

This hands-on experience prepares you for the next lesson in the track on Helm — where you'll learn how to package and deploy Python apps (including StatefulSets) into production more repeatably.