Liveness Probes on Python Health Endpoints

Your Python service is running in Kubernetes, but what happens when it hits a deadlock, a memory leak, or an infinite loop that makes it unresponsive? Without a liveness probe, Kubernetes will keep sending traffic to a zombie pod, and your users will experience timeouts and errors that are nearly impossible to diagnose. In this lesson, you'll learn how to configure liveness probes on Python health endpoints so Kubernetes can automatically detect and restart unhealthy pods, keeping your service resilient and your users happy.

The problem this lesson solves

Imagine you've deployed a Python Flask API that serves thousands of requests per minute. Everything works perfectly — until one day, a subtle bug in a background thread causes the main event loop to block permanently. The process is still running (so the container hasn't crashed), but it's no longer responding to any HTTP requests. Kubernetes, by default, only checks if the container process is alive — not whether it's actually working. So the pod stays in Running state, and all incoming traffic gets stuck waiting for a response that never comes.

This is the classic zombie pod problem. It's worse than a crash: a crash is detectable and restartable, but a hung process is invisible unless you explicitly probe it. The solution is a liveness probe — a health check that Kubernetes runs periodically to decide whether the container should be restarted. Python services are particularly prone to this kind of failure because of GIL deadlocks, thread exhaustion, or unhandled exceptions that leave the app in a broken but non-terminated state.

By the end of this lesson, you'll be able to:

• Add a health endpoint to your Python service that accurately reflects its liveness
• Configure a liveness probe in your Kubernetes Deployment spec
• Troubleshoot common probe failures and know when a liveness probe is the right tool (vs. a readiness or startup probe)

Core concept / mental model

Think of a liveness probe as a heartbeat monitor for your container. Kubernetes sends a request (typically HTTP GET) to a designated endpoint at a regular interval. If the endpoint returns a success status code (2xx or 3xx), everything is fine. If it fails (or doesn't respond in time), Kubernetes kills the container and replaces it with a new one.

Here's the key distinction: liveness is about is the process alive and responsive? It is not about is it ready to serve traffic? That's the job of a readiness probe. A deadlocks or corrupted state triggers a liveness failure, but a service that's still starting up or overloaded but not yet dead should not be restarted — that's where readiness probes shine.

To make a liveness probe useful, you need a health endpoint in your Python app that reports "I am alive" in a way that reflects more than just "the process hasn't crashed." A good health endpoint should:

• Be quick (return in milliseconds, not seconds)
• Avoid heavy I/O (no database queries, no complex computations) unless they're essential to liveness
• Return a clear success/failure status code (200 for healthy, 500 for unhealthy)
• Be independent of external dependencies (if a database is down, is your app really dead? probably not)

Analogy: Liveness probe = "Are you breathing?" Readiness probe = "Are you ready to take on new patients?" Startup probe = "Are you still warming up before I even start checking?"

How it works step by step

Setting up a liveness probe involves three layers, and each one builds on the previous:

1. Expose a health endpoint in your Python app — You need a simple HTTP route that returns 200 if the app is healthy, 500 if not.
2. Configure the probe in your Kubernetes Deployment — You add a livenessProbe block to your container spec, specifying the path, port, and timing parameters.
3. Deploy and observe — After deploying, Kubernetes will monitor the endpoint, and if it starts failing, it will restart the container automatically.

Let's break down each step.

Step 1: Design the health endpoint

The health endpoint should reflect actual liveness. In most cases, it's enough to simply check that the app's main thread is responsive. But if you have critical background workers, you might want to expose a flag that those workers update periodically.

Step 2: Choose the probe type

• httpGet: Most common for Python web apps. Kubernetes sends an HTTP GET to a specific path.
• tcpSocket: Checks if a TCP port is open — useful if you don't have HTTP, but less powerful.
• exec: Runs a command inside the container. Rarely needed for Python HTTP services.

Step 3: Set sensible timing parameters

• initialDelaySeconds: Give your app time to start up. If this is too short, Kubernetes might restart a pod that's just booting.
• periodSeconds: How often to probe. Default is 10s.
• timeoutSeconds: How long to wait for a response. Default is 1s.
• failureThreshold: Number of consecutive failures before restarting (default 3).
• successThreshold: Number of successes to consider healthy (default 1).

Pro tip: For a Python app that takes a few seconds to start (e.g., loads a large model or connects to a database), set initialDelaySeconds to at least this startup time — otherwise you'll see crash loops that are hard to debug.

Step 4: Watch the logs and events

If a liveness probe fails, Kubernetes will restart the container, and you'll see kubectl describe pod showing Liveness probe failed: HTTP probe failed with statuscode: 500. Check your app logs to see why the health endpoint returned 500.

Hands-on walkthrough

Let's put this into practice. We'll create a simple Flask app with a health endpoint, containerize it, and configure a liveness probe.

1. Create a Python health endpoint

Create app.py:

from flask import Flask, jsonify
import random

app = Flask(__name__)

# Simulate a failure condition - in real life this could be a stuck thread or deadlock
is_healthy = True

def background_worker():
    """Simulate a background task that might crash the app."""
    global is_healthy
    # Imagine this worker gets stuck and sets is_healthy to False
    # In a real app, you'd have actual logic here.
    pass

@app.route('/health')
def health():
    if is_healthy:
        return jsonify(status='ok'), 200
    else:
        return jsonify(status='degraded'), 500

@app.route('/')
def index():
    return 'Hello, World!'

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=8000)

2. Dockerfile

FROM python:3.11-slim

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY app.py .

EXPOSE 8000

CMD ["python", "app.py"]

requirements.txt:

flask==3.0.0

3. Deployment with liveness probe

deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: flask-app
spec:
  selector:
    matchLabels:
      app: flask
  template:
    metadata:
      labels:
        app: flask
    spec:
      containers:
        - name: flask-app
          image: your-registry/flask-app:latest
          ports:
            - containerPort: 8000
          livenessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 5
            periodSeconds: 10
            timeoutSeconds: 2
            failureThreshold: 3

4. Deploy and test

docker build -t your-registry/flask-app:latest .
docker push your-registry/flask-app:latest
kubectl apply -f deployment.yaml
kubectl get pods

Now, if your app's /health endpoint starts returning 500 (maybe you toggle is_healthy to False), Kubernetes will restart the container after 3 consecutive failures.

Compare options / when to choose what

When to use a liveness probe vs. readiness probe:

• Use liveness when your app can get into a state where it must be restarted (deadlock, infinite loop, memory leak).
• Use readiness when your app might be temporarily unavailable (loading data, warm-up, high load) but shouldn't be restarted.
• Use both together for the best resilience.

Pro tip: Don't make your liveness probe depend on external services (like a database). If the database goes down, your app might still be able to recover once the DB is back, but a liveness failure would kill it unnecessarily.

Troubleshooting & edge cases

Crash loop due to short initialDelaySeconds

Symptom: Pod keeps restarting, and kubectl logs shows that the app was just starting up when it was killed.

Fix: Increase initialDelaySeconds to cover your startup time, or add a startup probe with a longer period.

Liveness probe returns 500 for a transient issue

Symptom: A temporary spike in CPU or a slow database call makes /health take too long, and the probe times out (via timeoutSeconds) — leading to a restart.

Fix: Make sure your health endpoint returns quickly and doesn't do heavy work. Also increase timeoutSeconds to a value that accounts for GC pauses, but keep it low enough to detect a true hang.

Health endpoint is wrong (e.g., /health returns 404)

Symptom: Pod is restarted even though the app works for users. kubectl describe pod says HTTP probe failed with statuscode: 404.

Fix: Verify the path exists in your app, and test with curl inside the container: kubectl exec <pod> -- curl http://localhost:8000/health.

Your app uses threads that get stuck

Symptom: The main thread is responsive, but background workers are dead.

Fix: Have the health endpoint check a shared flag that workers update (e.g., through a heartbeat). If the flag is stale, return 500.

What you learned & what's next

In this lesson, you learned how to perform liveness probes on Python health endpoints. You now understand:

• Why liveness probes are crucial for detecting hung or deadlocked Python processes
• How to set up a health endpoint in a Python app that truly reflects liveness
• How to configure httpGet liveness probes in your Deployment manifest
• How to distinguish liveness from readiness and when to use which
• How to troubleshoot common probe failures

This knowledge is essential for running production-grade Python services on Kubernetes. Your next step is to explore readiness probes in the next lesson, or dive into startup probes for slow-starting applications like those that load large ML models. For a deeper dive, check out the official Kubernetes documentation on liveness probes.

Common Mistakes

• Ignoring initialDelaySeconds: Setting it to 0 or too low causes crash loops during startup.
• Making liveness depend on external services: If your health endpoint includes a DB check, a DB outage will restart your pod unnecessarily.
• Checking everything in the health endpoint: Adding heavy computations makes the probe slow and can cause false failures.
• Using readiness instead of liveness for hung services: A readiness probe marks the pod as not ready, but it never restarts it — the pod stays broken forever.

Variations

• Using FastAPI or Django: The pattern is the same, just adapt the health endpoint to your framework (e.g., a simple @app.get("/health") in FastAPI).
• Using Kubernetes startup probes: Use a startup probe to cover long startup times, then let liveness take over — this avoids tweaking initialDelaySeconds.
• Using exec probes for non-HTTP Python scripts: If your Python script isn't a web server, you can run a custom script that checks some condition and exits with a non-zero code.

Real-World Use Cases

• Flask API at a startup that hits a deadlock due to a non-thread-safe third-party library, causing all requests to hang — a liveness probe catches it and auto-restarts.
• FastAPI machine learning service that slowly accumulates memory leaks; the liveness probe detects when the service stops responding and replaces the container before OOMKill.
• Django background task worker (with Celery) that gets stuck in an infinite loop; an implementation health endpoint returns 500, and the probe restarts the worker.

Key Takeaways

• Liveness probes are essential for detecting and restarting hung Python processes in Kubernetes.
• A good health endpoint should be fast, check only what's necessary, and return correct status codes.
• Use httpGet probes for Python HTTP services; it's the most straightforward approach.
• Properly set initialDelaySeconds and failureThreshold to avoid restart loops.
• Liveness and readiness serve different purposes — use them together for best resilience.
• Always test your health endpoint manually before configuring the probe to avoid surprises.

Practice Recap

Try this: modify the Flask app from this lesson to simulate a stuck background thread (e.g., a flag that goes False), deploy it to a local cluster (like minikube or kind), and observe how Kubernetes restarts the pod when the liveness probe fails. Experiment with different failureThreshold values and see how it changes the behavior. This hands-on exercise will solidify your understanding before moving on to readiness probes.