Graceful Shutdown in Kubernetes

Your Python service is running fine in Kubernetes until the day you roll out a new version, and suddenly you see connection resets, dropped messages, or failed health checks. The culprit? The pod received a termination signal but didn't have time to finish its in-flight work — it was killed mid-request. In this lesson, you'll master the art of handling Python graceful shutdown in Kubernetes, ensuring your services drain connections, complete tasks, and exit cleanly every time. This is a critical skill for any Python developer running services in production, and it's the key to zero-downtime deployments.

The problem this lesson solves

When Kubernetes decides to terminate a pod — whether for a rolling update, node maintenance, or resource pressure — it sends a SIGTERM signal to the main process inside the container. Your Python application has a grace period (default 30 seconds) to shut down gracefully. If it ignores the signal or fails to finish within that window, Kubernetes sends SIGKILL, which forcibly kills the process.

For a web server like Flask or FastAPI, this can mean:

• In-flight HTTP requests get cut off — clients see errors or truncated responses.
• Background tasks (e.g., Celery workers, message consumers) lose data — messages are lost or marked as failed incorrectly.
• Shared resources (database connections, lock files, temp files) are left in an inconsistent state — causing corruption or blocking on startup.

The default behavior of Python is to terminate immediately on SIGTERM, which is rarely what you want. Without a graceful shutdown handler, you're gambling with data integrity and user experience every time you deploy.

Core concept / mental model

Think of a pod as a worker in a warehouse. When the manager (Kubernetes) says "your shift is over" (SIGTERM), a good worker finishes the current task, puts tools away, and clocks out. A bad worker just drops everything and runs out, leaving a mess for the next shift.

In technical terms, graceful shutdown is a coordination protocol between Kubernetes and your application:

1. Kubernetes sends SIGTERM to the main process (PID 1).
2. Your application catches the signal and begins a shutdown sequence.
3. It stops accepting new work (e.g., stops taking new requests or pulling from a queue).
4. It finishes in-flight work — completes HTTP responses, commits transactions, or acknowledges messages.
5. It cleans up resources — closes database pools, flushes logs, releases locks.
6. It exits with code 0, signaling a clean termination.

If the pod doesn't exit before the terminationGracePeriodSeconds (default 30), Kubernetes escalates to SIGKILL. The key is to make your Python app listen for SIGTERM and orchestrate a graceful shutdown.

How it works step by step

Step 1: Understand signal handling in Python

Python's signal module lets you register handlers for Unix signals. The most common for shutdown is SIGTERM. In a Flask or FastAPI app, you often run a development server with app.run(), which doesn't automatically handle graceful shutdown. Production servers like Gunicorn or uvicorn provide their own mechanisms, but you still need to ensure your app's background tasks shut down cleanly.

Step 2: Design your shutdown sequence

Identify what needs to happen during shutdown:

• Web server: Stop accepting new connections, wait for active requests to complete.
• Background workers: Stop pulling new tasks, finish the current one, and send an acknowledgment.
• Resource cleanup: Close database connections, cancel pending timers, flush caches.

Step 3: Register a signal handler

Here's a minimal pattern using signal.signal:

import signal
import time
import sys

def handle_sigterm(signum, frame):
    print("Received SIGTERM, shutting down gracefully...")
    # Perform cleanup here
    cleanup_database_connections()
    sys.exit(0)

signal.signal(signal.SIGTERM, handle_sigterm)

# Keep the process alive
while True:
    time.sleep(1)

This catches SIGTERM and calls your cleanup function before exiting. For a simple script, that's enough. For a full service, you need to integrate with your server's lifecycle.

Step 4: Use Kubernetes lifecycle hooks

Kubernetes provides preStop hooks that run before the SIGTERM is sent. This is useful for… well, actually, the preStop hook runs before SIGTERM, so it's not for graceful shutdown itself, but it can be used to delay termination (e.g., to drain connections from a load balancer). Many teams use a preStop hook with a sleep to give the load balancer time to remove the pod from the rotation. For example:

lifecycle:
  preStop:
    exec:
      command: ["sh", "-c", "sleep 5"]

This delays termination by 5 seconds, allowing external traffic to stop. You still need the app to handle SIGTERM afterward.

Hands-on walkthrough

Let's build a realistic example: a FastAPI app that gracefully shuts down when Kubernetes sends SIGTERM.

First, create a simple application (app.py):

# app.py
import asyncio
import signal
from fastapi import FastAPI
from contextlib import asynccontextmanager

app = FastAPI()

# Simulate background task
async def background_task():
    while True:
        print("Doing work...")
        await asyncio.sleep(1)

@asynccontextmanager
async def lifespan(app):
    task = asyncio.create_task(background_task())
    yield
    print("Shutting down gracefully...")
    task.cancel()
    try:
        await task
    except asyncio.CancelledError:
        print("Background task cancelled")
    print("Cleanup complete")

app.router.lifespan_context = lifespan

@app.get("/")
async def root():
    return {"message": "Hello, Kubernetes!"}

Now run it with uvicorn, which handles SIGTERM itself, but our lifespan will also run:

uvicorn app:app --host 0.0.0.0 --port 8000

When you send the process a SIGTERM (e.g., kill -TERM <pid>), you'll see:

INFO:     Shutting down
Shutting down gracefully...
Background task cancelled
Cleanup complete

Great, but in Kubernetes, we also need to set the terminationGracePeriodSeconds and consider the pod's shutdown order.

Kubernetes deployment example

Here's a deployment manifest with a 30-second grace period:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: python-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: python-app
  template:
    metadata:
      labels:
        app: python-app
    spec:
      terminationGracePeriodSeconds: 30
      containers:
      - name: app
        image: myregistry/python-app:1.0
        ports:
        - containerPort: 8000
        lifecycle:
          preStop:
            exec:
              command: ["sh", "-c", "sleep 5"]

Now, when a pod is terminated: 1. Kubernetes sends SIGTERM after a 5-second preStop sleep (if configured). 2. Your app catches SIGTERM, starts graceful shutdown, and finishes within 30 seconds. 3. If it doesn't, SIGKILL is sent.

Compare options / when to choose what

Different Python servers have built-in graceful shutdown mechanisms. Here's a comparison:

For background workers, use a flag to indicate shutdown. For example:

import signal
import time
import sys

shutdown_requested = False

def handle(signum, frame):
    global shutdown_requested
    shutdown_requested = True

signal.signal(signal.SIGTERM, handle)

while not shutdown_requested:
    # Poll queue or do work
    process_next_task()
    time.sleep(0.1)

print("Finished current task, shutting down")
sys.exit(0)

Troubleshooting & edge cases

• Pod still killed even with graceful shutdown — Check if your app is stuck waiting for a long-running request. Increase terminationGracePeriodSeconds or implement timeouts on your HTTP server (e.g., uvicorn's --timeout-keep-alive).
• Signal not received — If your container runs a shell script that starts Python, the shell may catch the signal. Use exec so Python becomes PID 1: exec python app.py.
• Background threads not stopping — If you use threads (e.g., threading), they won't be killed automatically. Use a threading.Event to signal threads to stop.
• preStop hook misuse — preStop runs before SIGTERM, so if you add a long sleep, you're delaying the shutdown, not making it graceful. Use it only to give load balancers time to remove the pod.

Pro tip: Always test your graceful shutdown locally by sending SIGTERM to your process and observing the logs. Then deploy to a test cluster and simulate pod termination with kubectl delete pod --grace-period=5 --force to see how it behaves.

What you learned & what's next

You now understand how to handle Python graceful shutdown in Kubernetes: you learned the importance of catching SIGTERM, orchestrating a clean shutdown sequence, and configuring Kubernetes with appropriate grace periods and lifecycle hooks. You've completed a hands-on exercise, connecting the concept to a real FastAPI application.

Next, in the Kubernetes for Python Developers track, you'll explore probes and readiness checks — how Kubernetes determines if your app is ready to serve traffic, which complements graceful shutdown perfectly. Keep building on this foundation to make your Python services bulletproof in the cloud.