PersistentVolume for Python Logs

Your Python service is writing logs like there's no tomorrow — and then the pod restarts. In seconds, those logs vanish into the void. That's the pain this lesson solves: by creating a Kubernetes PersistentVolume for Python logs, you ensure that critical application output survives pod restarts, rescheduling, and even node failures, giving you durable, inspection-ready logs when you need them most.

The problem this lesson solves

Containers are ephemeral by design. When your Python app writes to /tmp or even /var/log/myapp.log, that data lives inside the pod's ephemeral filesystem — which is wiped the moment the pod dies, restarts, or is evicted. In Kubernetes, pod restarts are routine: deployments roll out new images, nodes drain for maintenance, and liveness probes kill unresponsive containers. Every one of those events turns yesterday's error traces into ghost logs.

This isn't just about debugging after an incident. Audit trails, security events, and business metrics often live side by side with app logs, and losing them can have real consequences. The solution is persistence — storing logs on durable storage that outlives the pod. Kubernetes gives you that with PersistentVolumes (PVs) and PersistentVolumeClaims (PVCs).

Why Python specifically? Python's logging module writes to plain files just fine, but the container filesystem disappears with the pod. Pairing a standard Python logging FileHandler with a mounted PV transforms your logs from throwaway to durable — no code changes required.

Core concept / mental model

Think of a PersistentVolume as a shared, network-attached hard drive that Kubernetes can plug into any pod. A PersistentVolumeClaim is the request for storage — like asking the cluster's storage orchestrator for '50 Gi of fast disk.' The volume itself is provisioned independently of any pod, so even if a pod is deleted, the volume (and the logs on it) remain.

To visualize it, imagine a parking garage: - The PersistentVolume is a reserved parking spot — the actual resource (disk space, network storage) exists independently. - The PersistentVolumeClaim is your ticket that reserves that spot. - The Pod is the car that parks in it — it can come and go, but the spot remains.

In Kubernetes terms, the lifecycle works like this: 1. An admin (or dynamic provisioner) creates a PV bound to physical storage. 2. A PVC is created to claim that PV. 3. A pod mounts the PVC at a specific path. 4. The Python app writes logs to that path. 5. When the pod dies, the PVC and PV remain — and a new pod can mount the same volume and read the old logs.

How it works step by step

Creating a PersistentVolume for Python logs involves four layers, each with a specific job:

1. PersistentVolume (PV) — a cluster-level resource that points to actual storage (e.g., an NFS share, EBS volume, or local disk). This is the physical resource.
2. PersistentVolumeClaim (PVC) — a namespaced request for storage. Kubernetes binds it to a matching PV automatically.
3. Pod / Deployment — the workload that consumes the PVC via a volume mount.
4. Python app — writes logs to the mounted path, often using logging.FileHandler.

A typical flow for log persistence looks like this:

1. Create a PV YAML that defines storage capacity, access mode, and storage class.
2. Create a PVC that requests storage (size, access mode, storage class).
3. Deploy your Python pod with a volume mount referencing the PVC.
4. Your Python code writes log files to /var/log/app/ inside the mount.
5. After a pod restart, the logs are still there because the PV persists.

For this lesson, we'll use a hostPath PV — perfect for a single-node test cluster (like minikube or kind). It maps a node's filesystem path to a volume, giving you real persistence with zero cloud costs.

Hands-on walkthrough

Let's build a working example. Step 1: define a PersistentVolume that stores logs on the node's disk.

Create a file pv-log.yaml:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-python-logs
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  hostPath:
    path: /mnt/data/python-logs

Apply it:

kubectl apply -f pv-log.yaml
kubectl get pv

Output:

NAME             CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
pv-python-logs   1Gi        RWO            Retain           Available                       5s

Step 2: create a PVC that claims the volume.

Create pvc-log.yaml:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-python-logs
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 500Mi

Apply it:

kubectl apply -f pvc-log.yaml
kubectl get pvc

Output:

NAME              STATUS   VOLUME           CAPACITY   ACCESS MODES   STORAGECLASS   AGE
pvc-python-logs   Bound    pv-python-logs   1Gi        RWO                           6s

The status Bound confirms the PVC matched your PV.

Step 3: create a Python deployment that mounts the volume. We'll use a simple logging script that writes a log line every 5 seconds.

Create app.py:

import logging
import time

logging.basicConfig(
    filename='/var/log/app/app.log',
    level=logging.INFO,
    format='%(asctime)s - %(levelname)s - %(message)s'
)

while True:
    logging.info("Python log entry from persistent volume")
    time.sleep(5)

Create a Dockerfile:

FROM python:3.11-slim
COPY app.py /app.py
RUN mkdir -p /var/log/app
CMD ["python", "/app.py"]

Build and push the image (or use a local registry for testing). Then create deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: python-log-pod
spec:
  replicas: 1
  selector:
    matchLabels:
      app: python-log-pod
  template:
    metadata:
      labels:
        app: python-log-pod
    spec:
      containers:
      - name: logger
        image: your-registry/python-logger:1.0
        volumeMounts:
        - name: log-volume
          mountPath: /var/log/app
      volumes:
      - name: log-volume
        persistentVolumeClaim:
          claimName: pvc-python-logs

Deploy and verify:

kubectl apply -f deployment.yaml
kubectl get pods

Wait until the pod is running, then check the logs from inside the pod:

kubectl exec deploy/python-log-pod -- cat /var/log/app/app.log

Expected output (your timestamps will differ):

2025-01-15 14:32:01,213 - INFO - Python log entry from persistent volume
2025-01-15 14:32:06,215 - INFO - Python log entry from persistent volume

Now the moment of truth: delete the pod and watch the logs survive.

kubectl delete pod python-log-pod-<random-id>
kubectl get pods  # a new pod gets created automatically

Once the new pod is ready, check the logs again:

kubectl exec deploy/python-log-pod -- tail -n 2 /var/log/app/app.log

You'll see the log entries from before the pod was deleted. Congratulations — you've created a PersistentVolume for Python logs!

Compare options / when to choose what

Your choice of PV type depends on your environment. Here's a comparison:

For Python log volumes, start with hostPath in dev, then move to a cloud volume or NFS for multi-node production. The PVC interface stays the same — only the PV spec changes.

Troubleshooting & edge cases

The most common issue is a PVC stuck in Pending status. This means no PV matched your request. Check your PV's accessModes and capacity. For instance, asking for ReadWriteOnce (RWO) but your PV only supports ReadWriteMany (RWX) will fail. Also, PVCs are namespaced — you must create the PVC in the same namespace as your pod. If your PV has a storageClassName set and your PVC doesn't, they won't match. For hostPath, use storageClassName: "" on both if you have a default StorageClass in your cluster.

Another gotcha: the mount path must exist in the container image. My Dockerfile uses RUN mkdir -p /var/log/app to ensure it's there. If you mount to a non-existent path, the container will fail to start with Error: stat /var/log/app: no such file or directory.

Reference: every object needs a valid apiVersion. For PV, PVC, and Deployment, use v1. Don't use extensions/v1beta1 — that's deprecated and will break on modern clusters.

What you learned & what's next

You now understand the core idea: a PersistentVolume provides durable storage independent of pod lifecycles, and a PersistentVolumeClaim binds your pod to that storage. You've applied this by creating a PV for Python logs, wiring it through a PVC, and confirming logs survive pod deletion. You can scale this pattern to other stateful data like databases or file uploads.

Next in the track, you'll learn how to use StatefulSets and Headless Services for stateful Python applications — such as databases that need stable network identities. That's the natural progression from PVCs, since StatefulSets work hand-in-hand with persistent storage.

Keep experimenting: try changing the reclaim policy to Delete and see what happens to the PV, or mount the PV in two pods at once (with RWX) to test shared log access.