Run Python Stateful Apps with StatefulSets

You built a Python API, containerized it, and rolled it out with a Deployment — but what happens when your app needs to remember things? Maybe it's a worker that processes jobs from a queue and stores results locally, or a leader election service that must keep a stable identity. If you tried to use a plain Deployment, you'd quickly hit a wall: pods get random names, scale up and down unpredictably, and can't reliably attach to the same storage. That's the pain this lesson solves. You're about to learn how StatefulSets give your Python apps ordered identity, stable networking, and persistent storage — the missing piece for running stateful workloads like databases, caches, and message brokers in Kubernetes.

The problem this lesson solves

Imagine you have a Python application that records events to a local file or SQLite database. You deploy it as a Deployment with three replicas. Every time a pod restarts, it gets a new name like my-app-7d9f6b4c5-xk2pq. If another pod joins or leaves, your app's clients have no way to know which instance they're talking to. More importantly, when a pod is rescheduled to a different node, its local storage disappears. Your data is gone.

This is the fundamental problem: Deployments treat every pod as interchangeable and ephemeral. For stateless apps (web frontends, API servers) that's perfect. But for stateful apps — databases, caches, queues, or any Python service that needs to persist data or maintain a unique identity — you need something different.

StatefulSets solve this by giving each pod a stable identity and the ability to mount persistent storage that follows it. Instead of random names, you get mydb-0, mydb-1, mydb-2. Each pod knows its own name, its peers, and can attach to the same PersistentVolume even after a restart.

Core concept / mental model

Think of a Deployment like a fleet of identical delivery vans. Each van is interchangeable — if one breaks down, you just send another. But what if each van has a specific route and a lockbox that only that van can open? Then you need something like a StatefulSet: a convoy where every vehicle has a fixed number, its own route, and its own unlockable storage.

StatefulSets are purpose-built for distributed stateful applications. They provide:

• Stable network identity: Each pod gets a unique, persistent hostname like redis-0.redis.default.svc.cluster.local. Even if the pod is deleted and recreated, it keeps the same name.
• Stable persistent storage: Each pod is bound to a dedicated PersistentVolumeClaim (PVC). The storage survives pod restarts, rescheduling, and even deletion.
• Ordered deployment and scaling: Pods are created one at a time, in order from 0 to N-1, and are terminated in reverse order. This is critical for apps that need a leader (pod 0) to be ready before followers.
• Stable service endpoints: You can create a headless Service that exposes each pod individually, so clients can connect directly to db-0 or db-1.

The role of PersistentVolumes

A StatefulSet without persistent storage is just a Deployment with stable names. The real power comes from combining it with PersistentVolumes (PVs) and PersistentVolumeClaims (PVCs). Each replica gets its own PVC, defined in the volumeClaimTemplates spec. This is a template that Kubernetes uses to create a new PVC for each pod.

Headless services and stable DNS

To take full advantage of stable identities, you usually create a headless Service (one with clusterIP: None). This lets you query DNS for the list of pod IPs, and gives each pod a predictable DNS name based on its ordinal index.

How it works step by step

Let's break down the anatomy of a StatefulSet spec. Here's a minimal YAML for a Python-based stateful app (we'll use a simple FastAPI service that writes to a local file).

# statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: py-counter
spec:
  serviceName: py-counter
  replicas: 3
  selector:
    matchLabels:
      app: py-counter
  template:
    metadata:
      labels:
        app: py-counter
    spec:
      containers:
      - name: app
        image: python:3.11-slim
        command: ["/bin/sh", "-c"]
        args:
          - |
            pip install fastapi uvicorn
            echo 'from fastapi import FastAPI
            app = FastAPI()
            @app.get("/")
            def read_root():
                with open("/data/count.txt", "a") as f:
                    f.write("hit\n")
                return {"pod": "py-counter-0", "count": sum(1 for _ in open("/data/count.txt"))}' > /app/main.py
            uvicorn main:app --host 0.0.0.0 --port 8000
        ports:
        - containerPort: 8000
        volumeMounts:
        - name: data
          mountPath: /data
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 1Gi

The critical parts:

1. serviceName — must match a headless Service name; this gives each pod a DNS entry.
2. volumeClaimTemplates — defines a PVC template. Kubernetes will create data-py-counter-0, data-py-counter-1, etc.
3. podManagementPolicy — defaults to OrderedReady, meaning pods start sequentially.
4. replicas — the number of pods, each with its own stable identity.

When you apply this YAML, Kubernetes:

• Creates a PVC for each replica (if not already existing).
• Creates py-counter-0 first, waits for it to be Ready, then creates py-counter-1, and so on.
• When scaling down, it removes py-counter-2 first, then py-counter-1, then py-counter-0.

Hands-on walkthrough

Let's get your hands dirty. You'll need a running cluster (minikube or kind will do) and kubectl installed.

Step 1: Create a headless Service

The headless service provides the stable DNS names for your StatefulSet pods.

# headless-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: py-counter
spec:
  clusterIP: None
  selector:
    app: py-counter
  ports:
  - port: 8000
    targetPort: 8000

Apply it with kubectl apply -f headless-service.yaml.

Step 2: Deploy your StatefulSet

Now apply the StatefulSet YAML from above:

kubectl apply -f statefulset.yaml

Watch the pods come up one by one:

kubectl get pods -w

You should see output similar to:

py-counter-0   1/1   Running   0   10s
py-counter-1   0/1   Pending   0   2s
py-counter-1   1/1   Running   0   8s
py-counter-2   0/1   Pending   0   1s
py-counter-2   1/1   Running   0   7s

Notice the order: pod 0 is ready before pod 1 starts.

Step 3: Verify stable identity and storage

Check that each pod has its own PVC:

kubectl get pvc

You'll see data-py-counter-0, data-py-counter-1, and data-py-counter-2.

Now, write a small Python script to test that each pod has a unique identity. You can exec into a pod and call the API:

kubectl exec py-counter-0 -- curl -s localhost:8000/

You'll get a response like {"pod": "py-counter-0", "count": 1}. Each pod has its own file, so the counts are independent.

Step 4: Simulate a pod crash

Delete py-counter-1 and watch it come back with the same name and the same data:

kubectl delete pod py-counter-1
kubectl get pods -w

When it restarts, it will still be py-counter-1, and the data will still be there because it reattaches the same PVC.

Compare options / when to choose what

You might be wondering: when should I use a Deployment, a StatefulSet, or a DaemonSet? Here's a quick comparison:

Alternatives to StatefulSets

• Deployments with external storage: For some apps, you can store state in an external database (like PostgreSQL) or object storage (like S3) and keep the pod stateless. This is simpler and more flexible, but adds network latency and operational complexity.
• Operator pattern: Tools like the PostgreSQL Operator or the Kafka Operator use custom controllers to manage stateful applications, often on top of StatefulSets. They handle backups, failover, and scaling automagically.
• Helm charts: Many stateful apps (e.g., Redis, MySQL) come with Helm charts that wrap StatefulSets in a convenient package, handling configuration and upgrades.

When to choose StatefulSets

Choose StatefulSets when:

• Your app needs persistent storage that survives pod restarts.
• You need stable DNS names for peer discovery (e.g., in a distributed system).
• You need ordered startup/shutdown (e.g., a primary database before read replicas).
• Your Python app is a datastore, cache, or queue — like Redis, Elasticsearch, or RabbitMQ.

Avoid them for purely stateless applications where Deployments are simpler and faster to scale.

Troubleshooting & edge cases

StatefulSets can be tricky. Here are common problems and how to fix them.

Pod stuck in Pending

If your pod stays Pending, it likely can't bind a PVC. Check:

kubectl describe pod py-counter-0
kubectl get pvc

The error will often say waiting for a volume to be created. This usually means the cluster doesn't have a StorageClass that can provision volumes automatically. You need to either install a storage provisioner (like local-path for kind) or pre-create PVs.

PVC stuck in Pending

Same root cause. Check your StorageClass:

kubectl get storageclass

If none exists, you might need to use hostPath for development, or install the standard storage provisioner. For minikube, you can enable the default storage class:

minikube addons enable storage-provisioner

Scaling down loses data (surprise!)

If you scale down a StatefulSet to 0 replicas, the PVCs remain and the data is preserved. But if you delete the StatefulSet entirely (with kubectl delete statefulset), the PVCs are not deleted by default. That's good for data safety but can leave orphaned volumes. To clean up, you must delete the PVCs explicitly.

Stable identity doesn't mean stable IP

In Kubernetes, pod IPs can change, even with StatefulSets. The stable identity is the hostname, not the IP. Clients should use the DNS name (e.g., py-counter-1.py-counter.default.svc.cluster.local) rather than the IP address.

Ordered shutdown takes time

Because StatefulSets terminate pods in reverse order and wait for graceful termination, scaling down can be slower than with Deployments. If your app has long-running tasks, ensure it handles SIGTERM gracefully.

Pro tip: Always set terminationGracePeriodSeconds appropriately for your Python apps — especially if they need to flush data or close connections cleanly.

What you learned & what's next

You've just unlocked the power of StatefulSets for Python applications. You now know:

• Why StatefulSets exist: to give stable identity, storage, and ordered scaling to stateful services.
• How to build one: from a headless Service to volumeClaimTemplates.
• When to use them: for databases, caches, queues, and any Python service that must persist data.
• How to troubleshoot: from PVC issues to graceful shutdown.

This is a cornerstone of running production-grade Python backends on Kubernetes. In the next lesson, you'll learn how to manage and update StatefulSets — rolling out new versions of your stateful app without downtime. You'll also explore how to integrate them with Headless Services for service discovery, and how to use kubectl rollout commands specifically for StatefulSets.

Keep that PVC close, and your data will never be far behind.