Liveness Probes in Kubernetes

You've deployed a container to Kubernetes, and everything seems fine — until your application deadlocks, hangs, or enters an infinite loop. Without a liveness probe, Kubernetes assumes the container is healthy because its process is still running. Your users see timeouts or blank pages while your cluster remains blissfully unaware. Liveness probes solve exactly this problem: they give Kubernetes a way to ask your application "Are you alive?" and, if the answer is no, automatically restart the container without manual intervention.

The problem this lesson solves

Modern applications fail in subtle ways that a simple process-health check can't detect. A web server may be running but unable to serve requests due to a locked goroutine. A database connection pool may be exhausted, making every query hang. A background worker might consume 100% CPU but never make progress. In each case, the container's PID 1 is still alive, so kubectl get pods shows Running. But the service is effectively dead.

Liveness probes provide a deep health check — one that tests whether the application is truly functioning, not just whether its process exists. When a liveness probe fails repeatedly, Kubernetes restarts the container, potentially restoring service automatically. This is the difference between an application that silently degrades and one that self-heals.

The alternative — no probe — leads to cascading failures: pods become zombies, traffic is routed to broken backends, and SREs are paged at 3 AM. Using liveness probes is a fundamental reliability pattern in production Kubernetes clusters.

Core concept / mental model

Think of a liveness probe as a heartbeat monitor for your container. Just as a hospital monitor checks a patient's pulse at regular intervals, Kubernetes runs a small check on your container every few seconds. If the check fails repeatedly (configurable threshold), Kubernetes declares the container unhealthy and restarts it.

This is different from a readiness probe, which controls whether traffic is sent to the pod. Readiness probes say "Is this pod ready to receive requests?" while liveness probes say "Is this pod healthy enough to keep running?" A pod can be ready (serving traffic) but still fail a liveness probe if its internal state is corrupted.

Pro tip: Always use both liveness and readiness probes. Liveness probes restart deadlocked containers; readiness probes prevent sending traffic to pods that are temporarily overwhelmed or initializing.

Probe types available

Kubernetes supports three types of checks: - HTTP probe: Makes an HTTP GET request to a specific endpoint. Success = a 2xx or 3xx status code. - TCP probe: Attempts to open a TCP connection to a specified port. Success = connection established. - Exec probe: Runs a command inside the container. Success = exit code 0.

Each probe has configurable parameters: initialDelaySeconds (how long to wait before first check), periodSeconds (frequency), timeoutSeconds (max time for the check), failureThreshold (consecutive failures before restart), and successThreshold (consecutive successes after failure to mark healthy).

How it works step by step

1. Pod starts: Kubernetes creates the container and waits for initialDelaySeconds before the first probe. This gives the application time to initialize.
2. Probe executes: After the initial delay, Kubernetes runs the probe (HTTP, TCP, or exec) at every periodSeconds interval.
3. Success: If the probe succeeds, Kubernetes continues normally — no action taken.
4. Failure handling: If the probe fails once, Kubernetes increments a failure counter. It continues probing.
5. Threshold reached: When the failure counter reaches failureThreshold (default 3), Kubernetes triggers a container restart.
6. Restart: The pod's restartPolicy (default Always) causes a new container to be created. The process repeats from step 1.

Container states during probe lifecycle

• Running: Container is executing, but probes may not have started yet (initialDelaySeconds hasn't elapsed).
• Healthy: Probe succeeded — container is considered alive.
• Unhealthy: Probe has failed failureThreshold times — container will be terminated and restarted.
• CrashLoopBackOff: After repeated restarts, Kubernetes backs off exponentially (10s, 20s, 40s, etc.). This is normal; it prevents rapid restart cycles.

Hands-on walkthrough

Let's create two deployments — one with a working liveness probe, and one that simulates a deadlock to see how probes kick in.

Example 1: HTTP liveness probe on a simple web server

Create a file liveness-http.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: liveness-http
spec:
  replicas: 1
  selector:
    matchLabels:
      app: liveness-http
  template:
    metadata:
      labels:
        app: liveness-http
    spec:
      containers:
      - name: nginx
        image: nginx:1.25
        ports:
        - containerPort: 80
        livenessProbe:
          httpGet:
            path: /healthz
            port: 80
          initialDelaySeconds: 3
          periodSeconds: 5

Apply it:

kubectl apply -f liveness-http.yaml
kubectl get pods -w

Because nginx doesn't have a /healthz endpoint, the probe will fail. Watch the pod get restarted every ~15 seconds (3 failures * 5 seconds). You'll see RESTARTS increment in kubectl get pods.

Expected output: The pod enters a CrashLoopBackOff state after a few restarts because the probe always fails.

Example 2: Exec liveness probe with a health script

Create a deployment that checks a marker file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: liveness-exec
spec:
  replicas: 1
  selector:
    matchLabels:
      app: liveness-exec
  template:
    metadata:
      labels:
        app: liveness-exec
    spec:
      containers:
      - name: busybox
        image: busybox:1.36
        args:
        - /bin/sh
        - -c
        - |
          touch /tmp/healthy
          sleep 30
          rm /tmp/healthy
          sleep 600
        livenessProbe:
          exec:
            command:
            - cat
            - /tmp/healthy
          initialDelaySeconds: 5
          periodSeconds: 5

Apply and watch:

kubectl apply -f liveness-exec.yaml
kubectl get pods -w

After 30 seconds, the container removes /tmp/healthy. The exec probe (cat /tmp/healthy) fails, and after failureThreshold failures, the container restarts. You'll see the RESTARTS count increase.

Example 3: TCP liveness probe for a database

apiVersion: apps/v1
kind: Deployment
metadata:
  name: liveness-tcp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: liveness-tcp
  template:
    metadata:
      labels:
        app: liveness-tcp
    spec:
      containers:
      - name: redis
        image: redis:7.2
        ports:
        - containerPort: 6379
        livenessProbe:
          tcpSocket:
            port: 6379
          initialDelaySeconds: 5
          periodSeconds: 10

kubectl apply -f liveness-tcp.yaml
kubectl get pods

This probe simply checks that Redis is listening on port 6379. It's a lightweight health check, but it won't detect if Redis has deadlocked while still accepting TCP connections.

Compare options / when to choose what

Recommendation: Start with HTTP probes if your app has an HTTP endpoint. Use TCP for infrastructure components like Redis or MySQL where HTTP would add unnecessary complexity. Use exec only when you need to run custom logic inside the container.

Troubleshooting & edge cases

Common mistakes

• Initial delay too short: The probe fires before the app starts, causing unnecessary restarts. Set initialDelaySeconds to at least your app's startup time.
• Overusing liveness probes: If a pod restarts too frequently, it may be in CrashLoopBackOff. Tune failureThreshold and periodSeconds to avoid flapping.
• Confusing liveness and readiness: A readiness probe can temporarily remove a pod from service without restarting it; a liveness probe restarts it. Using only liveness probes means all recovery is through restarts.
• Slow probes: If your HTTP handler takes too long, the probe times out (default 1 second). Increase timeoutSeconds or optimize the endpoint.

Edge cases

• Startup probe: For slow-starting applications (e.g., Java apps), Kubernetes 1.18+ offers startup probes that run only during initialization, giving the app time to start before liveness probes take over.
• Deadlock detection: A liveness probe that checks the same deadlock is useless. Example: a /healthz endpoint that returns 200 even when the worker pool is locked. Design probes to test actual functionality (e.g., run a simple query, check a worker queue).
• Probe configuration drift: Copy-paste liveness probe config without tuning it for each service. Always adjust initialDelaySeconds and periodSeconds based on your app's behavior.

What you learned & what's next

You now understand how liveness probes give Kubernetes eyes into your container's true health. You've seen HTTP, TCP, and exec probes in action, learned their trade-offs, and practiced configuring restart policies. The key takeaway: a liveness probe is your first line of defense against silent application failures.

In the next lesson, you'll learn about readiness probes — which control traffic routing rather than restarts — and how to combine both for a production-grade deployment.

Pro tip: In production, always define a liveness probe AND a readiness probe. The liveness probe restarts unhealthy containers; the readiness probe keeps traffic away from pods that are still initializing or temporarily overloaded.

Ready to master readiness probes? Continue to the next lesson in the Kubernetes track.