Graceful Shutdown in Python Pods

You've built a Python service, containerized it, and deployed it to Kubernetes. Everything works — until you roll out a new version or scale down. Users hit connection reset by peer, in-flight requests get truncated, and your logs show a mess of unhandled exceptions. The root cause? Your pod is being killed instantly by Kubernetes without a chance to finish its work. This is the pain that graceful shutdown solves, and in this lesson you'll learn how to implement it in your Python pods so your services go down cleanly, every time.

The problem this lesson solves

Kubernetes is a ruthless but predictable orchestrator. When a pod needs to be terminated — during a rolling update, a scale-down, or a node drain — Kubernetes sends a SIGTERM signal to the main process (PID 1) of each container. It then waits silently for a grace period (default 30 seconds) before sending SIGKILL.

If your Python app ignores SIGTERM, it keeps running until the grace period expires, and then Kubernetes force-kills it. The result: open connections are dropped, in-flight requests are lost, and clients see errors like Connection reset by peer or Broken pipe. For stateful services or APIs handling critical data, this is unacceptable.

Implement graceful shutdown in Python pods means your application catches SIGTERM, stops accepting new requests, finishes processing the ones already in progress, and then exits cleanly within the grace period. This lesson gives you a production-ready pattern.

The problem is real: a 2023 survey showed that over 40% of production incidents in microservices involve abrupt termination. Kubernetes gives you a 30-second window by default — you must use it wisely.

Core concept / mental model

Think of your pod as a busy restaurant kitchen. When the manager (Kubernetes) decides to close for the night, they don't just flip off the lights — they tell the staff (your app) they're closing, wait for the last customers to finish their meals, clean up, and then lock the door. That's graceful shutdown. Flipping off the lights is the equivalent of SIGKILL.

Key terms:

• SIGTERM — a polite "please stop" signal (15 on most systems). Your app can catch and handle it.
• SIGKILL — an unignorable "you stop now" signal. Your app cannot catch it.
• terminationGracePeriodSeconds — a pod spec field that controls how long Kubernetes waits between SIGTERM and SIGKILL. Default is 30s.
• PreStop hook — a container hook that lets you run a command or HTTP request before the main process receives SIGTERM.

Here's a conceptual diagram of the shutdown sequence:

Kubernetes → Pod termination requested
  → PreStop hook runs (if defined)
  → SIGTERM sent to PID 1
  → App catches SIGTERM, starts draining
  → App finishes in-flight requests, closes connections
  → App exits (code 0)
  → If grace period expires first → SIGKILL

Why it matters for Python: Python's default signal handling for SIGTERM is to terminate the process immediately. If you use gunicorn, uvicorn, or other WSGI/ASGI servers, they may have their own shutdown logic, but you still need to coordinate your own cleanup (e.g., closing database connections, flushing metrics).

How it works step by step

The graceful shutdown process follows a predictable sequence. Here's how to implement it in Python:

1. Receive SIGTERM — Use Python's signal module or a library like uvicorn's built-in handlers to catch the signal.
2. Stop accepting new work — Set a flag (e.g., shutdown_event) that your server checks. If you're using a web framework, put it in maintenance mode or use a callback to reject new connections.
3. Finish in-flight tasks — Wait for all active requests/connections to complete. This may involve joining worker threads or waiting on a queue.
4. Clean up resources — Close database connections, flush logs/metrics, release locks.
5. Exit with code 0 — Signal a successful shutdown so Kubernetes doesn't think it crashed.
6. Tune the grace period — Set terminationGracePeriodSeconds in your pod spec to match your app's typical cleanup time.

For Kubernetes, you also have the PreStop hook option — a script that runs before SIGTERM is sent. It's useful for tasks like deregistering from a service mesh or notifying a load balancer.

Hands-on walkthrough

Let's implement graceful shutdown in a real Python service. We'll start with a simple Flask app, then improve it with a health-check-aware pattern.

Example 1: Basic Flask app with SIGTERM handling

from flask import Flask, jsonify
import signal
import time
import threading
import sys

app = Flask(__name__)
shutdown_event = threading.Event()

@app.route("/health")
def health():
    return jsonify({"status": "ok"})

@app.route("/work")
def work():
    # Simulate a long-running task
    time.sleep(10)
    return jsonify({"done": True})

def handle_sigterm(signum, frame):
    print("Received SIGTERM, shutting down gracefully...")
    shutdown_event.set()

signal.signal(signal.SIGTERM, handle_sigterm)

if __name__ == "__main__":
    print("Starting Flask app...")
    # Run Flask in a separate thread so we can wait for shutdown
    def run_app():
        app.run(host="0.0.0.0", port=8080)

    thread = threading.Thread(target=run_app)
    thread.start()

    # Wait for shutdown signal
    shutdown_event.wait()
    print("Shutting down gracefully...")
    # Give Flask a moment to finish in-flight requests
    time.sleep(2)
    sys.exit(0)

Expected output: When you send SIGTERM (e.g., with kubectl delete pod) the app logs:

Received SIGTERM, shutting down gracefully...
Shutting down gracefully...

Example 2: Using uvicorn with Python's asyncio

For ASGI apps (FastAPI, Starlette), you can handle signals directly with asyncio:

import asyncio
import signal
import uvicorn

class GracefulServer:
    def __init__(self, app, host, port):
        self.app = app
        self.host = host
        self.port = port
        self.shutdown_event = asyncio.Event()

    async def handle_signal(self, sig, frame):
        print(f"Received {sig.name}, draining connections...")
        self.shutdown_event.set()

    async def main(self):
        loop = asyncio.get_running_loop()
        for sig in (signal.SIGINT, signal.SIGTERM):
            loop.add_signal_handler(sig, lambda s=sig: asyncio.create_task(self.handle_signal(s, None)))

        config = uvicorn.Config(self.app, host=self.host, port=self.port)
        server = uvicorn.Server(config)

        # Start server
        asyncio.create_task(server.serve())

        # Wait for shutdown event
        await self.shutdown_event.wait()
        print("Shutting down...")
        # Uvicorn will handle graceful shutdown of connections
        server.should_exit = True

# Use it with your FastAPI app
if __name__ == "__main__":
    from fastapi import FastAPI
    app = FastAPI()

    @app.get("/")
    async def root():
        return {"message": "Hello World"}

    graceful = GracefulServer(app, "0.0.0.0", 8000)
    asyncio.run(graceful.main())

Expected output: Send SIGTERM, and uvicorn logs a graceful shutdown, draining connections before exit.

Example 3: PreStop hook for advanced coordination

In your Kubernetes deployment, add a PreStop hook to deregister from a service mesh or load balancer:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-python-app
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: app
        image: my-python-app:latest
        ports:
        - containerPort: 8080
        terminationGracePeriodSeconds: 30
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh", "-c", "sleep 5"]

The sleep 5 gives the load balancer time to mark the pod as not-ready before the app gets SIGTERM. This is a common pattern to avoid dropping in-flight connections during rolling updates.

Compare options / when to choose what

There are several ways to implement graceful shutdown in Python. Here's a comparison:

Rule of thumb: For most Python microservices, start with uvicorn/gunicorn's built-in signal handling, then add your own cleanup logic using signal handlers. Use PreStop hooks only when you need to perform an action before the main process shuts down (e.g., deregistration).

Troubleshooting & edge cases

1. Pods stuck in Terminating state

If your pod remains in Terminating beyond the grace period, Kubernetes sends SIGKILL. This usually means your app doesn't handle SIGTERM or is blocking on a long-running task.

Fix: Ensure your signal handler is non-blocking and sets an event. In threaded apps, avoid blocking the main thread with time.sleep after receiving the signal.

2. SIGTERM not being caught

Common issue: running your app with python app.py often works, but if you use a shell script as the entrypoint, the shell may not forward signals. Also, if you're using gunicorn, it catches signals itself and may not propagate to worker processes.

Fix: Use exec in your Docker entrypoint to make the Python process PID 1:

CMD ["python", "app.py"]

Or use a proper process manager like tini.

3. In-flight requests dropped despite handling signals

If your server doesn't stop accepting new connections, it might still crash. For Flask's dev server, there's no automatic draining; you need to handle it manually or use gunicorn/uvicorn.

Fix: Use a production WSGI/ASGI server that supports graceful shutdown natively.

4. Grace period too short

If your app takes longer than terminationGracePeriodSeconds, it gets force-killed. Increase the pod spec value, but also optimize your shutdown logic.

5. Empty reply from server during rolling update

This often happens when the service is removed from the load balancer before it's ready to shut down. Add a PreStop hook with a short sleep (e.g., 5s) to allow the proxy to update its endpoints.

What you learned & what's next

You now understand how to implement graceful shutdown in Python pods:

• Explain the core idea: SIGTERM is a signal to drain, not kill. Your app must stop accepting work, finish current work, and exit cleanly within the grace period.
• Apply it in a practical exercise: You've used signal handlers with Flask, asyncio with uvicorn, and PreStop hooks.
• Connect to Kubernetes: You've seen how terminationGracePeriodSeconds and lifecycle hooks fit into the pod termination flow.

The knowledge ties directly to improving reliability of your deployments. In the next lesson, you'll learn how to handle pod startup with readiness probes — ensuring your service only receives traffic when it's ready. That pairs perfectly with graceful shutdown to make your service lifecycle bulletproof.

Keep practicing: modify the Flask example to handle in-flight requests explicitly, and experiment with different grace periods to find the sweet spot for your workload.