StatefulSet with Persistent Storage

You’ve mastered Deployments — they restart, scale, and update your app without breaking a sweat. But when your app needs a unique identity — a pod that comes back with the same name and the same disk data after a restart — a standard Deployment falls short. This is exactly where a StatefulSet shines, and in this lesson you’ll deploy a StatefulSet with persistent storage using PersistentVolumeClaims (PVCs) and volume claim templates, giving each pod its own private, surviving disk.

The problem this lesson solves

A standard Deployment treats pods as interchangeable cattle — they’re disposable, and their names are random. For stateless apps like a web frontend that pulls data from a shared database, that’s perfect. But consider a database cluster (like PostgreSQL, Cassandra, or MongoDB), a message queue (RabbitMQ), or a distributed file system — each pod needs:

• A stable, sticky identity (same hostname after restart)
• Persistent storage that survives pod crashes or reschedules
• Ordered creation, scaling, and termination (e.g., Pod-0 before Pod-1)

Without these guarantees, your app will lose data or get confused about which node is the primary and which is a replica. StatefulSet solves all three problems.

Core concept / mental model

Think of a StatefulSet as a numbered, named team where each member keeps their own locker.

• Stable network identity: Pods are named {statefulset-name}-{ordinal} — e.g., web-0, web-1, web-2. When a pod dies and is recreated, it gets the same name, and DNS records point to it again.
• Persistent storage per pod: A StatefulSet uses a volume claim template to create a dedicated PVC for each pod instance. Each pod gets its own folder on disk — Pod-0’s data is segregated from Pod-1’s.
• Ordered, graceful operations: Pods are created in increasing ordinal order (0 → 1 → 2) and terminated in reverse order (2 → 1 → 0). This is critical for database leader-election or quorum setups.

Key definitions

• Headless Service: A Service without a cluster IP (clusterIP: None). It gives each StatefulSet pod its own DNS record (e.g., web-0.web-svc.default.svc.cluster.local).
• Volume Claim Template: A blueprint inside the StatefulSet spec that tells Kubernetes: for each pod replica, create a PVC with these storage requirements and mount it.
• PersistentVolumeClaim (PVC): A request for storage. If a matching PersistentVolume (PV) exists or a dynamic provisioner is available, storage is automatically bound.

How it works step by step

When you kubectl apply -f statefulset.yaml, the following happens:

1. Headless Service is created (if defined). DNS entries for pod-name.svc-name become resolvable.
2. Pod-0 is created. The StatefulSet controller sees spec.replicas=3 and starts with ordinal 0.
3. Volume claim template triggers PVC creation. Pod-0’s spec references volumeClaimTemplates[n]. Kubernetes creates a PVC named {volume-claim-name}-{statefulset-name}-{ordinal} (e.g., www-web-0).
4. PVC binds to a PV. Either a pre-existing PV or a dynamically provisioned one matches the PVC’s storage class, access mode, and size. The PVC is owned by the StatefulSet and lives until the pod is deleted only if you set persistentVolumeReclaimPolicy: Delete.
5. The pod mounts the volume at the specified mountPath. Even if Pod-0 crashes and is rescheduled on a different node, the same PVC still exists, and the new pod mounts it.
6. Next pod created — only after Pod-0 reports Running and Ready. This ordered creation continues up to replicas-1.

Pro tip: If your storage provisioner supports it, set volumeClaimTemplates.spec.storageClassName to a class with volumeBindingMode: WaitForFirstConsumer. This delays PV binding until the pod is scheduled, ensuring the PV is created in the same zone as the pod.

Hands-on walkthrough

Let’s deploy a single-replica PostgreSQL StatefulSet with persistent storage. You can adapt this to any app that needs stable identity and disk.

1. Create a Headless Service

# postgres-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: postgres
  labels:
    app: postgres
spec:
  clusterIP: None  # headless — gives each pod its own DNS
  ports:
    - port: 5432
      name: postgres
  selector:
    app: postgres

kubectl apply -f postgres-service.yaml

2. Create the StatefulSet with Volume Claim Template

# postgres-statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  serviceName: postgres
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
        - name: postgres
          image: postgres:15
          env:
            - name: POSTGRES_PASSWORD
              value: secret
          ports:
            - containerPort: 5432
              name: postgres
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 10Gi

kubectl apply -f postgres-statefulset.yaml

3. Verify the StatefulSet

kubectl get statefulset postgres
# NAME       READY   AGE
# postgres   1/1     30s

kubectl get pods
# NAME         READY   STATUS    RESTARTS   AGE
# postgres-0   1/1     Running   0          30s

kubectl get pvc
# NAME             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
# data-postgres-0  Bound    pvc-abc123...                              10Gi       RWO            standard       30s

Notice the PVC is named data-postgres-0 — that’s {volumeClaimTemplate.name}-{statefulset.name}-{ordinal}. If you scale the StatefulSet to replicas: 2, you get data-postgres-1 for postgres-1.

4. Test persistence

Write some data, delete the pod, and see it reappear with the same content.

# Write test data
kubectl exec postgres-0 -- bash -c "echo 'persistent data' > /tmp/test.txt"

# Delete the pod
kubectl delete pod postgres-0

# New pod is created with the same name and PVC
kubectl wait --for=condition=ready pod/postgres-0

# Check the data is still there
kubectl exec postgres-0 -- cat /tmp/test.txt
# persistent data

Pro tip: If your app writes to a subdirectory inside the volume, be aware that some base images (like official Postgres) already initialize the data directory. You may need to set a different mountPath or use subPath if your app expects an empty directory.

Compare options / when to choose what

• Use StatefulSet when each pod needs its own data directory and a stable hostname. Examples: PostgreSQL primary/replica, Cassandra, Elasticsearch, ZooKeeper.
• Use Deployment with a shared PVC (e.g., ReadWriteMany) when all pods read/write the same data — like a WordPress site sharing uploaded files.
• Use Operator (like Strimzi for Kafka, Zalando for PostgreSQL) for production-grade stateful apps — they handle backups, scaling, and cluster reconfiguration beyond what StatefulSet offers.

Troubleshooting & edge cases

PVC stuck in Pending

kubectl describe pvc data-postgres-0

If you see waiting for a volume to be created, either by external provisioner or manually, ensure: - A StorageClass exists and is configured with a dynamic provisioner (kubectl get storageclass). - Your nodes have sufficient disk capacity for the requested size. - If using volumeBindingMode: WaitForFirstConsumer, the pod must be scheduled first — check pod events.

Pod stuck in Pending after PVC binds

Check the pod events:

kubectl describe pod postgres-0

Common causes: - The PVC’s access mode ReadWriteOnce means the volume can be mounted on only one node. If two pods of the same StatefulSet try to use the same PVC, the second pod will fail. In a multi-replica StatefulSet, each pod must have its own PVC — which the volume claim template guarantees. - The PV was provisioned in a different availability zone than the pod’s node. Use WaitForFirstConsumer to avoid this.

Postgres-specific: data directory not empty

The official Postgres image refuses to start if the mounted volume already contains a lost+found or other files. Fix by adding an emptyDir init container or using a subPath:

volumeMounts:
  - name: data
    mountPath: /var/lib/postgresql/data
    subPath: pgdata

What you learned & what's next

You now know how to deploy a StatefulSet with persistent storage — you understand: - The fundamental difference between StatefulSet and Deployment for stateful workloads - How volume claim templates create per-pod PVCs - How ordered pod management works and why it matters for databases - How to verify persistent data survives pod deletion

This is the foundational pattern for running stateful applications in production. In the next lesson, you’ll learn how to update and roll back a StatefulSet without data loss, and then move on to Headless Services and DNS for connecting pods in a cluster.

Your next topic: Rolling updates for StatefulSets.