Troubleshoot Python Pod Crash Loops

Your Python service is scaling beautifully in Kubernetes, but suddenly every pod shows CrashLoopBackOff and the deployment is bouncing faster than a rubber ball. You're not alone—crash loops are one of the most common and frustrating issues Python developers face when moving to Kubernetes, especially when your app worked perfectly in local Docker. This lesson gives you a battle-tested, step-by-step approach to troubleshoot python pod crash loops: how to inspect logs, events, and restart policies, and how to apply targeted fixes so your pods stay healthy and your users stay happy.

The Problem This Lesson Solves

When a pod crashes and restarts repeatedly, Kubernetes doesn't just give up—it enters a CrashLoopBackOff state, backing off with increasing delays between restarts. For a Python developer, this usually means a code-level issue: an unhandled exception on startup, a missing dependency, a bad environment variable, or a resource limit that's too tight. Left unchecked, crash loops can cause prolonged downtime, especially if your Deployment has a low replicas count.

What makes crash loops particularly tricky is that Kubernetes hides the root cause behind layers of abstraction: the pod spec, the container runtime, the logs, and the cluster events. Without a systematic method, you'll waste hours guessing. This lesson arms you with a repeatable diagnostic process—starting from the health of the pod up, down to your Python code—so you can find and fix the issue quickly, even under pressure.

Core Concept / Mental Model

Think of a crash loop like a car that won't start: the symptoms (engine clicking) are obvious, but the cause could be a dead battery, a jammed starter, or an empty fuel tank. In Kubernetes, the pod is the car, the container is the engine, and the restart policy is the ignition system that keeps trying to turn the key.

Here’s the mental model you need:

• Liveness probe: Kubernetes checks if the app is still alive by hitting an endpoint (e.g., /health). If it fails, the kubelet kills the container and restarts it.
• Readiness probe: This only affects service routing, not restarts. A failing readiness probe removes the pod from service endpoints but does not restart it.
• Restart policy: The default for Deployments is Always. So any container exit (even a normal one) triggers a restart—which is often the root of the loop.
• Exit codes: Each container exit has a numeric code: 0 means success, 1 means generic error, 137 means killed by SIGKILL (often OOM), 143 means SIGTERM.

The crash loop is not a single failure—it’s a cycle: start → crash → backoff → restart → start again. Your goal is to break that cycle by addressing the root cause.

To remember this, use the WHEEL mnemonic: Watch logs, Help (diagnose) events, Examine exit codes, Evaluate probes, Look at resources (limits).

How It Works Step by Step

Diagnosing a crash loop is a structured exercise. Follow these steps in order—don’t skip ahead.

Step 1: Identify the Pod and Container Name

Use kubectl get pods to list all pods in the namespace. Look for one in CrashLoopBackOff. Note the pod name and the container name (usually the same if you have a single container).

Step 2: Inspect Logs for the Last Crash

Logs are your best friend. Use kubectl logs <pod> --previous to see the logs from the container that crashed—the current container may not have written anything yet. If you don’t see the --previous flag, you’ll often get logs from the crashing attempt, but mixing them up can mislead you.

Step 3: Check Events and Exit Codes

Run kubectl describe pod <pod> and look for the Events section. Pay attention to messages like Back-off restarting failed container, OOMKilled, or Failed to pull image. Also note the Last State and its Reason—e.g., Error with exit code 1, or OOMKilled with code 137.

Step 4: Validate Resource Limits

Check the pod spec for CPU/memory limits. A container that exceeds its memory limit gets killed by the kernel (OOMKilled). Use kubectl get pod <pod> -o yaml to see the resources section.

Step 5: Review Probes

If liveness probe is defined, verify the endpoint path and port. A probe that’s misconfigured (e.g., pointing to /healthz when your app only has /health) will cause repeated kills, even if your app is perfectly healthy.

Step 6: Reproduce Locally

Once you suspect code-level issues, run the same container image locally with docker run. Test with the same environment variables and startup commands. This isolates whether it’s a code problem or a cluster-configuration problem.

Step 7: Apply a Fix and Roll Out

Once you’ve identified the root cause, patch the Deployment—either by editing the YAML or using kubectl set env. Then roll out with kubectl rollout restart deployment <name> and verify.

Hands-On Walkthrough

Let’s apply the steps with a real example. Suppose you have a Python Flask app that crashes because a required environment variable is missing.

Example 1: Missing Environment Variable

Create a simple pod manifest that references an undefined env var:

apiVersion: v1
kind: Pod
metadata:
  name: crash-python
spec:
  containers:
  - name: app
    image: python:3.11-slim
    command: ["/bin/sh", "-c"]
    args:
      - |
        python -c "import os; print(os.environ['REQUIRED_VAR'])"
    restartPolicy: OnFailure

Apply it and watch the loop:

kubectl apply -f crash-pod.yaml
kubectl get pods
# NAME         READY   STATUS             RESTARTS   AGE
# crash-python 0/1     CrashLoopBackOff   3          45s

Now check logs:

kubectl logs crash-python --previous
# Traceback (most recent call last):
#   File "<string>", line 1, in <module>
#   File "/usr/local/lib/python3.11/os.py", line 679, in __getitem__
#     raise KeyError(key) from None
# KeyError: 'REQUIRED_VAR'

The log clearly shows a KeyError. Fix it by setting the env var in the manifest.

Example 2: Memory Limit Exceeded

Suppose your Python service uses a lot of memory. Create a Deployment with a memory limit of 64Mi:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mem-hog
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mem-hog
  template:
    metadata:
      labels:
        app: mem-hog
    spec:
      containers:
      - name: app
        image: python:3.11-slim
        command: ["python", "-c", "x = ' ' * (100*1024*1024); import time; time.sleep(60)"]
        resources:
          limits:
            memory: 64Mi

Apply and check events:

kubectl apply -f mem-hog-deploy.yaml
kubectl get pods
kubectl describe pod mem-hog-xxx
# Last State: Terminated
# Reason: OOMKilled
# Exit Code: 137

The exit code 137 and OOMKilled tell you the limit is too low. Increase the memory limit or optimize your Python code.

Example 3: Liveness Probe Misconfiguration

Here’s a Flask app where the probe hits the wrong path:

from flask import Flask
app = Flask(__name__)

@app.route("/health")
def health():
    return "OK"

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=5000)

The manifest sets the liveness probe to /healthz:

livenessProbe:
  httpGet:
    path: /healthz
    port: 5000
  initialDelaySeconds: 5
  periodSeconds: 5

The probe fails, so Kubernetes kills the container. Logs look fine, but describe shows liveness probe failures. Fix the path to /health—problem solved.

Compare Options / When to Choose What

There are different ways to diagnose and resolve crash loops. Here’s a comparison of common approaches:

When to choose what: For a first look, use kubectl logs --previous to catch stack traces. If the logs are empty or unclear, describe will reveal OOM or probe issues. For persistent issues, adopt centralized logging so you don’t lose crash logs between restarts.

Troubleshooting & Edge Cases

Even with a systematic approach, you’ll hit edge cases. Here are the most common and how to fix them.

Python-specific pitfalls

• Unhandled exceptions at import time: If a dependency isn’t installed or a config error occurs at import, the app crashes before you can log anything. Wrap your startup in try/except and log the traceback early.
• Missing if __name__ == '__main__' guard: If you use multiprocessing, the child processes re-import the module and might execute startup code repeatedly, causing loops.
• Wrong entrypoint: Ensure your Dockerfile’s CMD matches your Python executable. A typo like python app.py instead of python ./app.py can cause immediate exits.

Kubernetes configuration errors

• Readiness vs liveness probe mix-up: A failing readiness probe doesn’t restart, but causes the pod to be removed from service. It might look like a crash if you have zero replicas ready. Double-check which probe you changed.
• Image pull issues: A wrong image tag or a private registry without credentials causes ErrImagePull/ImagePullBackOff, which is often confused with a crash loop. Check the events for Failed to pull image.
• Restart policy on Pods: If you’re using a standalone Pod (not a Deployment), the default restartPolicy is Never, so you may see a Failed state instead of a loop. Deployments always set it to Always.

Fix common issues

• OOMKilled: Increase resources.limits.memory, or reduce memory usage in Python (e.g., use gc, stream large data). Monitor with kubectl top pod.
• Probe failure: Fix the endpoint path or adjust initialDelaySeconds—your app might need more time to start.
• Env var missing: Use ConfigMaps/Secrets and validate them in code with a clear error message instead of a KeyError.

What You Learned & What's Next

You now have a reliable, step-by-step method to troubleshoot python pod crash loops: check --previous logs, inspect events and exit codes, validate resource limits and probes, reproduce locally, and apply targeted fixes. You learned how to interpret exit codes like 137 (OOMKilled) and why the restart policy matters in Kubernetes. You’ve practiced real exercises with missing env vars, memory overruns, and probe misconfigurations.

Next, you’ll build on this by exploring how to automate crash-loop detection with monitoring and alerts—so you’re not manually watching pods. That lesson will cover tools like Prometheus, custom liveness probes, and Python-based operators that can watch and recover from failures automatically.