Rollback Python Deployments

You've just pushed a new version of your Python service to Kubernetes, and suddenly your API latency spikes, your error rate climbs, and support tickets start flooding in. Your pipeline greenlit the deployment, but production disagrees. In the old world, you'd scramble to revert code and redeploy, praying the fix lands fast. With Kubernetes, you have a superpower: instant, controlled rollback. This lesson teaches you exactly how to handle rollbacks for Python deployments using kubectl rollout — the fastest way to recover from a bad release and keep your service healthy.

The problem this lesson solves

Deploying is not the finish line; it's a step on a path that can go sideways. In production, a Python service can fail silently due to a subtle dependency change, a misconfigured environment variable, or a data migration bug that only surfaces under real traffic. Rollbacks are your safety net. Without a clear strategy, a bad deployment means prolonged downtime, manual code reverts, and a stressed-on-call engineer.

You've likely used version control to revert code, but Kubernetes introduces a new layer: deployment revisions. Every change to your Deployment creates a new revision, and Kubernetes lets you switch back to a previous one instantly. This lesson gives you a repeatable, low-risk way to handle rollbacks for Python deployments, whether you're running a simple Flask API or a distributed Celery worker fleet.

Core concept / mental model

Think of a Kubernetes Deployment as a time machine with a log of successful states. Each time you apply a change — a new image tag, changed replicas, new environment variables — Kubernetes records that as a new revision. The deployment controller's job is to move the cluster from the current revision to the desired one, updating Pods in a controlled rollout.

A rollback is simply telling the controller: "Go back to a previous revision." This is like using git revert but without the merge conflicts — Kubernetes handles the transition for you, following the same rollout strategy (e.g., RollingUpdate) in reverse.

A few key terms:

• Rollout: The process of updating Pods to match a new Deployment spec.
• Revision: An immutable snapshot of the Deployment's spec at a given point in time.
• Rollback: Moving the Deployment spec back to a previous revision.
• Rollout history: The list of all revisions your Deployment has gone through.

A helpful analogy: your Deployment is like a video game save system. You have multiple save slots (revisions). When you realize a new level (release) is buggy, you can load an earlier save and keep playing. Kubernetes keeps those saves for you.

How it works step by step

When you run kubectl rollout undo, Kubernetes performs these steps:

1. Look up the target revision (or the previous one, if you don't specify).
2. Update the Deployment's pod template to match that revision's spec.
3. Trigger a new rollout, just like a normal update.
4. Apply the configured strategy — by default RollingUpdate, so it creates new healthy Pods before terminating old ones.
5. Mark the rollout as complete when all replicas match the desired state.

This means a rollback is not a magic switch — it's a structured deployment in reverse, with the same safety checks (readiness probes, minReadySeconds) to avoid downtime.

You also need to be aware of revisions:

• Every change to .spec.template (the pod spec) creates a new revision.
• Scaling replicas or changing labels does not create a new revision.
• The rollout history is kept according to revisionHistoryLimit (default 10); older revisions are pruned, so you can't roll back beyond that.

Pro tip: Always check your rollout history before you need it. Knowing your revisions beforehand makes rollback decisions faster and less panicky.

Hands-on walkthrough

Let's practice with a real Python API. We'll deploy a simple Flask app, update it to a broken version, and roll back to the working one.

Step 1: Create the Deployment

Save the following as flask-deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: flask-api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: flask-api
  template:
    metadata:
      labels:
        app: flask-api
    spec:
      containers:
      - name: flask
        image: myregistry/flask-api:1.0.0
        ports:
        - containerPort: 5000
        readinessProbe:
          httpGet:
            path: /health
            port: 5000
          initialDelaySeconds: 3
          periodSeconds: 5

Deploy it and check the rollout status:

kubectl apply -f flask-deployment.yaml
kubectl rollout status deployment/flask-api

Expected output (truncated):

deployment "flask-api" successfully rolled out

Step 2: Update to a new version

Now update the image to 2.0.0 and roll it out:

kubectl set image deployment/flask-api flask=myregistry/flask-api:2.0.0
kubectl rollout status deployment/flask-api

The deployment should be successful. Check the history:

kubectl rollout history deployment/flask-api

Expected output:

REVISION  CHANGE-CAUSE
1         <none>
2         <none>

Pro tip: Use kubectl apply with a --record flag or annotation to see what caused each revision (e.g., kubectl apply -f deployment.yaml --record=true). This makes history far more readable.

Step 3: Break the new version

Imagine version 2.0.0 crashes. Simulate this by setting a container that exits immediately:

kubectl set image deployment/flask-api flask=myregistry/flask-api:2.0.0-broken

Check the rollout status — it will hang because pods fail readiness probes. Press Ctrl+C to stop waiting.

Step 4: Roll back to the previous revision

Now the moment of truth:

kubectl rollout undo deployment/flask-api

This reverts to the previous revision (revision 1). Verify the rollout:

kubectl rollout status deployment/flask-api
kubectl get pods -l app=flask-api

All pods should be Running and Ready again.

Rolling back to a specific revision

If you have three or more revisions and need to go back to exactly revision 1, specify it:

kubectl rollout undo deployment/flask-api --to-revision=1

This gives you precise control when you need to skip ahead, not just go one step back.

Compare options / when to choose what

Kubernetes offers several ways to handle bad deployments. Here's how they compare:

When to choose what:

• For a quick recovery from a recent bad deployment, use kubectl rollout undo — it's fast and doesn't require extra context.
• If you know a specific version was stable, roll back to that exact revision.
• If your Deployment spec changed (e.g., you need to revert environment variables), consider re-applying the old YAML file — but be aware that creates a new revision, not a true historical rollback.
• For code-level bugs, fix the code, build a new image, and deploy forward — rolling back only addresses the symptom.

Pro tip: Treat rollback as a temporary emergency measure. Always follow up with a proper fix and a new deployment, then delete the faulty revision from history if needed (though Kubernetes doesn't easily let you delete revisions — they're pruned automatically).

Troubleshooting & edge cases

Even rollbacks can go wrong. Here are common issues and how to fix them:

• Rollback hangs or times out: Pods may not be becoming ready. Check kubectl get events and kubectl describe pod — often the readiness probe fails on the rolled-back version too. Consider adjusting your probes.
• Can't rollback because revision not found: Your revisionHistoryLimit is too low. The default is 10, but if you've made many changes, older revisions are gone. Increase revisionHistoryLimit in your Deployment spec.
• Rollback makes things worse: The previous revision might rely on a ConfigMap or Secret that has already changed. Rollback only reverts the pod template, not your other resources. Double-check your ConfigMaps and Secrets.
• Accidentally rolling back to the same revision: If you use --to-revision with the current revision, nothing changes. Verify with kubectl rollout status.
• CPU/memory spike during rollback: RollingUpdate creates new pods before killing old ones, doubling resource usage temporarily. Ensure your cluster has headroom, or switch to strategy.type: Recreate for simpler rollbacks (though that causes downtime).
• Pod stuck in CrashLoopBackOff: The container is starting but crashing immediately. Check logs (kubectl logs <pod>) — often a missing environment variable or broken dependency.

What you learned & what's next

You now have the knowledge and hands-on skill to handle rollbacks for Python deployments with confidence. You understand the concept of revisions, the kubectl rollout undo workflow, how to check status and history, and how to troubleshoot common failure points. This is a critical safety skill for any production Python service on Kubernetes.

In the next lesson, you'll build on this by learning how to make your deployments even more resilient — possibly with health checks, autoscaling, or advanced deployment strategies. Each new skill layers onto the last, making you a more effective Kubernetes operator for your Python applications.

Now go practice: create a deployment, break it intentionally, and roll it back. The more you rehearse this, the less panic you'll feel when a real incident happens.