Roll Back a Python Deployment

You've just pushed v2.4.1 of your Python service to Kubernetes. Your kubectl rollout status deployment/myapp shows deployment \"myapp\" successfully rolled out — but your users are staring at 500 errors. Your CI passed, your tests were green, and yet production is broken. This is the reality of deploying software: rollouts fail, and the ability to roll back a broken Python deployment quickly and safely is what separates a panic-stricken engineer from a confident one. In this lesson, you'll learn how to use Kubernetes rollback mechanisms to revert to a known-good state, automate the process with the Python client, and avoid the common pitfalls that turn a simple rollback into a full-blown incident.

The problem this lesson solves

Deploying a new version of your Python application is exciting — until it isn't. A seemingly innocent change in your FastAPI or Django app can introduce a subtle bug that only manifests under production traffic. Without a fast and reliable rollback strategy, you face extended downtime, angry users, and a painful debugging session. The problem is not that rollouts fail; it's that many developers don't know how to roll back when they do. They resort to manually redeploying old images, editing Deployments with kubectl edit, or worse — reverting Git commits and going through the entire CI/CD pipeline again. These approaches are slow, error-prone, and often make the situation worse. Kubernetes provides a built-in solution: rollout history and kubectl rollout undo. This lesson will teach you to leverage these tools to restore service in seconds, not hours.

Core concept / mental model

Think of a Kubernetes Deployment as a manager for a fleet of identical pods running your Python code. When you update the Deployment's image, Kubernetes doesn't rip out all old pods at once. Instead, it performs a rolling update: it slowly replaces old pods with new ones, checking health along the way. This is like changing the tires on a moving car — you don't want to stop the car and lose momentum.

Each rollout is a revision in the Deployment's revision history. Whenever you change the pod template (image, environment variables, etc.), Kubernetes records it as a new revision. This history is your safety net. A rollback is simply telling Kubernetes to revert to a previous revision — to go back to a known-good state. The Deployment controller then orchestrates the reverse: it scales down the bad pods and scales up the pods from the earlier revision, following the same rolling update rules.

Imagine you have a versioned application like v2.4.0 (good) and v2.4.1 (bad). Kubernetes keeps these revisions in its memory (and in etcd). When you run kubectl rollout undo, Kubernetes reverts the pod template to the one from the chosen revision and creates a new rollout — effectively a new revision that mirrors the old one. This is not the same as reverting your Git history; it's an infrastructure-level revert that happens live.

How it works step by step

Let's walk through the lifecycle of a broken deployment and the rollback process. You'll see the cause-and-effect at each stage.

Prerequisites: Check rollout status

Before you can roll back, you need to know what state your Deployment is in. The first command is always kubectl rollout status:

kubectl rollout status deployment/myapp

If the rollout is still in progress, you'll see a message like Waiting for deployment \"myapp\" rollout to finish: 2 of 3 updated replicas are available.... If it's stuck, you'll see a timeout or an error. Use kubectl rollout history to list all revisions:

kubectl rollout history deployment/myapp

Each revision is a numbered 'step'. The output looks like:

deployments/"myapp"
REVISION  CHANGE-CAUSE
1         kubectl create --filename=myapp.yaml --record=true
2         kubectl edit deployment/myapp
3         kubectl set image deployment/myapp myapp=myapp:v2.4.1 --record=true

Roll back to the previous revision

If revision 3 is broken, you likely want to go back to revision 2. The straightforward command is:

kubectl rollout undo deployment/myapp

This rolls back to the previous revision (2). You can also roll back to a specific revision with --to-revision. For example:

kubectl rollout undo deployment/myapp --to-revision=2

The Deployment controller immediately starts a new rolling update to replace the bad pods. It follows the same update strategy (default RollingUpdate, with maxUnavailable and maxSurge). The rollback is not instantaneous — it takes time for new pods to become ready. But it's much faster and safer than manually deleting pods.

Verify the rollback

After the undo command, you must verify the rollout completed successfully:

kubectl rollout status deployment/myapp

You want to see: deployment \"myapp\" successfully rolled out. Then check the pod status and your application's health endpoint:

kubectl get pods
curl https://myapp.example.com/health

If the health endpoint returns 200, your rollback worked. If not, you may need to roll back further (e.g., to revision 1) or investigate deeper.

Hands-on walkthrough

It's time to practice with a real Python deployment. We'll use a simple Flask application and simulate a broken rollout. This walkthrough covers three main tasks: creating the initial Deployment, breaking it, and rolling back.

Set up a Python deployment

First, create a working directory and a Python file app.py that serves a health endpoint:

# app.py
from flask import Flask, jsonify
import os
import random

app = Flask(__name__)

@app.route(\"/health\")
def health():
    # Simulate: 10% chance of failure in the broken version
    if os.environ.get(\"VERSION\") == \"broken\" and random.random() < 0.1:
        return \"unhealthy\", 500
    return jsonify({\"status\": \"ok\", \"version\": os.environ.get(\"VERSION\", \"good\")})

if __name__ == \"__main__\":
    app.run(host=\"0.0.0.0\", port=8080)

Build two images, one good and one broken:

docker build -t myapp:good -f Dockerfile .
docker build -t myapp:broken -f Dockerfile .

(To simulate, we'll use environment variables instead of different code, but in practice you'd have different images.)

Create a Deployment deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: myapp
        image: myapp:good
        ports:
        - containerPort: 8080
        env:
        - name: VERSION
          value: \"good\"

Apply it and record the cause:

kubectl apply -f deployment.yaml --record=true
kubectl rollout status deployment/myapp

Break the deployment

Update the image to the broken version and set the env variable to trigger failures:

kubectl set image deployment/myapp myapp=myapp:broken
kubectl set env deployment/myapp VERSION=broken

Watch the rollout. It may succeed (the pods become Running), but if you test the app, some requests will fail. In a real scenario, you'd notice errors in your monitoring.

Roll back to the good revision

Check the rollout history to confirm revisions:

kubectl rollout history deployment/myapp

Then roll back to the previous good revision:

kubectl rollout undo deployment/myapp

Now verify the pods are running the good image and the health endpoint works. To do this automatically, we can use the Python client to check the status:

from kubernetes import client, config
import time

config.load_kube_config()
v1 = client.AppsV1Api()

# Wait for rollout to complete
def wait_for_rollout(name, namespace=\"default\"):
    for _ in range(60):
        deployment = v1.read_namespaced_deployment(name, namespace)
        if deployment.status.observed_generation == deployment.metadata.generation:
            if deployment.status.updated_replicas == deployment.status.replicas:
                if deployment.status.available_replicas == deployment.status.replicas:
                    return True
        time.sleep(2)
    return False

# Trigger rollback via kubectl? No, use the patch approach below.
# For simplicity, we assume you ran kubectl rollout undo.

print(\"Rollback complete\" if wait_for_rollout(\"myapp\") else \"Rollback timed out\")

Expected output:

Rollback complete

To automate the rollback entirely from Python, you can call the Kubernetes API to undo a rollout. The approach is to set the rollout.kubernetes.io/revision annotation to the previous revision, which triggers the Deployment controller to roll back. Here's a complete example:

from kubernetes import client, config

config.load_kube_config()
v1 = client.AppsV1Api()

def rollback_deployment(name, namespace=\"default\", revision=None):
    # Get current deployment
    deployment = v1.read_namespaced_deployment(name, namespace)
    # Get rollout history to find previous revision if not specified
    if revision is None:
        hist = v1.read_namespaced_deployment_status(name, namespace)
        # (Simplified: use the deployment generation as revision, but real history is more complex)
        revision = max(int(d.metadata.annotations.get(\"deployment.kubernetes.io/revision\", \"0\")) for d in [])
        # For production, use the history API via the client; here we'll just use the previous revision from the annotation.
    # Patch the deployment to trigger rollback
    patch = {
        \"metadata\": {
            \"annotations\": {
                \"deployment.kubernetes.io/revision\": str(revision)
            }
        }
    }
    v1.patch_namespaced_deployment(name, namespace, patch)
    print(f\"Rollback to revision {revision} triggered\")

# Example: rollback to revision 2
rollback_deployment(\"myapp\", revision=2)

Note: This is a simplified example; for production, use the kubectl rollout undo command or implement a proper controller using the kubernetes_asyncio client. The Python client is more useful for checking status and orchestrating multiple deployments.

Compare options / when to choose what

When your Python deployment breaks, you have several rollback strategies. The table below compares them:

When to use which

• kubectl rollout undo — Your first line of defense. It's fast, simple, and requires no knowledge of which revision was good. Use it when you immediately notice the rollout broke and you want to go back one step.
• --to-revision — When the last revision was also broken, or you want to skip intermediate versions. For example, if revision 3 broke and revision 2 also had a bug, you might go to revision 1.
• Manual image revert — If you accidentally scaled up the wrong image or you know exactly which image tag worked, this can be quicker than finding the revision number. However, it's more error-prone because you might forget other changes (env vars, etc.).
• Git revert + CI/CD — When the broken code contains a fundamental flaw that needs code changes (not just a bad image). This is the slowest but most permanent solution.
• Canary — In high-traffic environments, you might use a progressive delivery tool that automatically rolls back if the error rate spikes. This is beyond the scope of this lesson but is a powerful extension.

Troubleshooting & edge cases

Even kubectl rollout undo can fail. Here are common issues and how to fix them.

Rollout stuck in "Waiting" state

If kubectl rollout status hangs and never shows "successfully rolled out", your new pods might be failing health checks. The Deployment controller won't complete the rollout if maxUnavailable is 0 and the new pods never become ready. In this case, run:

kubectl get pods
kubectl logs <pod-name>

Look for the Python application crashing or not binding to the port. After seeing the error, you can force a rollback even if the rollout is incomplete:

kubectl rollout undo deployment/myapp

Kubernetes will interrupt the current rollout and start the rollback. If it still gets stuck, check your rolling update strategy. If maxUnavailable: 0, the controller waits for new pods to be ready; but if your app's readiness probe fails, it'll never proceed. A safer strategy for critical services is to set maxUnavailable: 1 to allow some old pods to shut down before new ones come up.

Rollback to a revision that doesn't exist

If you specify --to-revision=5 but your history only goes to 3, you'll get an error: error: unable to find specified revision 5 in history. Always check the history first with kubectl rollout history.

The rollout history is empty

If you created your Deployment without --record=true, the CHANGE-CAUSE column will be empty, but revisions still exist. To see full details of a revision, use

kubectl rollout history deployment/myapp --revision=2

Even without --record, you can still roll back. The --record flag is deprecated in favor of annotations, but it's still useful for tracking.

Error: "Deployment does not have any rollback information"

This happens when the Deployment was created through a different mechanism (e.g., a Helm chart) and the rollout history is not available. In this case, you can't use kubectl rollout undo. Instead, use helm rollback if you used Helm, or manually update the image. This is a good reason to standardize deployment methods.

What you learned & what's next

By now, you should confidently handle a broken Python deployment. You learned that a Deployment keeps a revision history, how to inspect it with kubectl rollout history, and how to revert with kubectl rollout undo. You also saw how to automate checks with the Python client, and you know the trade-offs between quick rollbacks and permanent code fixes.

You've achieved the learning objectives: - Explain the core idea behind rolling back a broken Python deployment: Kubernetes Deployment rollout history and the rollout undo command. - Complete a practical exercise: you simulated a broken Flask deployment and rolled it back using both kubectl and a Python script.

The next lesson in this track will cover Blue-green deployments, where you'll learn to switch traffic between two environments for even safer releases. You'll see how to combine rollback strategies with service routing for zero-downtime deployments.