Use Startup Probes

Your Python service takes 45 seconds to load a 2 GB machine-learning model into memory at boot — but Kubernetes marks your pod Ready after 10 seconds of liveness checks succeeding, then kills and restarts it in a crash loop before the model ever finishes loading. Liveness probes alone can't distinguish 'process started' from 'app is actually ready to serve traffic'. If you've ever watched a perfectly healthy Python API get perpetually restarted by Kubernetes for being 'slow', this lesson is for you: you'll learn how startup probes signal, 'Hey, I'm still booting — give me a minute before you judge my health.'

The problem this lesson solves

Kubernetes has two classic ways to check a container's health: liveness probes (is the process alive?) and readiness probes (can it receive traffic?). Both start executing as soon as the container starts — and that's the problem for slow-booting Python services.

Your Python app may be doing heavy initialization at boot:

• Loading a neural network or scikit-learn pickle (often hundreds of MB)
• Connecting to a database and running migrations
• Warming up a connection pool or caching layer
• Compiling JIT-heavy code (e.g., with numba or Cython)
• Downloading remote configuration or feature flags

If your liveness probe starts checking at second 5 and your app takes 30 seconds to be ready, the probe will fail and Kubernetes will kill the container — even though everything would have been fine if it had just waited a bit longer. The result: a CrashLoopBackOff with a perfectly healthy application.

The old workarounds were ugly: set huge initialDelaySeconds on liveness probes (fragile, guesswork), or wrap your app in a shell script that sleeps before starting (slow, brittle). Startup probes are the clean, native solution built into Kubernetes.

Core concept / mental model

Think of it like a three-stage system:

1. Startup probe — 'Are you still booting?' It runs during an initial period and, while it succeeds, Kubernetes won't run liveness or readiness probes.
2. Readiness probe — 'Are you ready to receive traffic?' It gates whether the pod gets added to service endpoints.
3. Liveness probe — 'Should we restart you?' If this fails after startup, Kubernetes kills the container.

A startup probe is a temporary gate: it lets your slow boot finish without being interrupted. Once it succeeds, the startup probe stops running, and the regular liveness/readiness probes take over. It's a one-time check for the boot phase.

Pro tip: Think of startup probe as a pass/fail entry exam — it doesn't judge performance, it just confirms the app has reached a baseline 'I'm up' state. After it passes, the regular health checks take over.

Key parameters to keep in mind:

• initialDelaySeconds — wait before first probe (default 0)
• periodSeconds — how often to probe (default 10)
• failureThreshold — how many consecutive failures until the container is killed (default 1)
• timeoutSeconds — per-probe timeout (default 1)

How it works step by step

When you add a startup probe to a container, here's the sequence:

1. Container starts and runs its entrypoint command.
2. Kubernetes waits initialDelaySeconds (if set).
3. It sends startup probe requests every periodSeconds.
4. If the probe succeeds, the startup probe is considered complete — Kubernetes marks the container as started and begins running liveness and readiness probes.
5. If the probe fails, Kubernetes retries up to failureThreshold times. If all fail, the container is killed and restarted (subject to restartPolicy).
6. After startup completes, the startup probe is no longer run — it only exists for the boot phase.

The startup probe doesn't affect readiness directly; it just delays when liveness and readiness begin. That's what saves your slow Python service from the restart loop.

Hands-on walkthrough

Let's create a realistic example: a FastAPI app that sleeps for 25 seconds to simulate model loading, then serves a health endpoint.

Step 1: Write the Python app

# app.py
import time
import uvicorn
from fastapi import FastAPI

app = FastAPI()

# Simulate slow initialization (e.g., loading a model)
print("Loading model...")
time.sleep(25)
print("Model loaded!")

@app.get("/health")
def health():
    return {"status": "ok"}

if __name__ == "__main__":
    uvicorn.run(app, host="0.0.0.0", port=8000)

Step 2: Containerize it

# Dockerfile
FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app.py .
CMD ["python", "app.py"]

requirements.txt:

fastapi
uvicorn

Step 3: Deploy with a startup probe

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: slow-python-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: slow-python-app
  template:
    metadata:
      labels:
        app: slow-python-app
    spec:
      containers:
      - name: app
        image: my-repo/slow-python-app:latest
        ports:
        - containerPort: 8000
        startupProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 5
          periodSeconds: 5
          failureThreshold: 10
        readinessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 0
          periodSeconds: 5
        livenessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 0
          periodSeconds: 10

Here's the math: the startup probe gives up to failureThreshold * periodSeconds = 10 * 5 = 50 seconds to succeed. Your app takes ~25 seconds plus startup time, so it passes comfortably. The liveness and readiness probes won't start until the startup probe succeeds — so they won't interfere during boot.

Step 4: Verify it works

Apply the deployment and watch the pod states:

kubectl apply -f deployment.yaml
kubectl get pods -w

You should see the pod go from Pending → Running → Ready. If you inspect the events, you'll see the startup probe running before readiness:

Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  12s   default-scheduler  Successfully assigned ...
  Normal  Pulling    11s   kubelet            Pulling image ...
  Normal  Started    10s   kubelet            Started container
  Normal  Created    10s   kubelet            Created container

The pod becomes Ready only after the startup probe passes and the readiness probe succeeds.

Compare options / when to choose what

So when should you use a startup probe? Whenever your Python app takes longer than a few seconds to be ready — typically because of model loads, DB connection delays, or caching warm-up. Use both readiness and liveness after startup for a robust setup.

Troubleshooting & edge cases

Here are the most common problems and how to fix them:

1. The startup probe never succeeds, causing CrashLoopBackOff

• Check your health endpoint: it must return 200 (or your expected status) during initialization. If the endpoint only exists after the heavy load, the probe will fail.
• Increase failureThreshold or periodSeconds to give more time. But don't set absurdly high values that mask genuine crashes.
• Look at logs and events: kubectl logs <pod>, kubectl describe pod <pod>.

2. The startup probe succeeds too quickly, then liveness kills the pod later

• This happens when your startup probe uses a generic endpoint (like a socket check) that passes before the app is actually ready. Make the startup probe hit the same HTTP endpoint as your readiness probe (e.g., /health), and ensure that it only returns success when the app is truly ready to serve.

3. initialDelaySeconds is set too low

• If your container needs a moment to start the HTTP server, set initialDelaySeconds to a few seconds (e.g., 5) to avoid connection refused errors.

4. Python process is still running but the probe fails

• Your app might be single-threaded and blocking during initialization. Use exec or tcpSocket probes if HTTP isn't available, or refactor your app to start the health server before the heavy load (e.g., using a background thread for model loading).

5. Startup probe with a command that doesn't exist

• If you use exec, ensure the command is available in the container (e.g., python -c "import requests; requests.get('http://localhost:8000/health')" might fail if requests isn't installed). Prefer httpGet when possible.

What you learned & what's next

You now understand how to use startup probes for slow-starting Python services — you can explain the core idea, apply it in a practical exercise, and connect it to broader Kubernetes health-check strategies. You can configure a startup probe with the right timing to prevent crash loops, and you know when to choose a startup probe over quick fixes like high initialDelaySeconds.

Next in the track, you'll likely explore readiness probes in more depth or move on to resource limits — but for now, make sure your slow-booting Python services are protected with a startup probe. Try this exercise: modify the example above to make the health endpoint return 503 while the app is still initializing, and then see how the startup probe behaves.

Pro tip: Always document the expected maximum startup time in your deployment manifests. Future you (or your team) will thank you when they tweak the model and wonder why the pod keeps restarting.