Roll back a faulty Deployment

You’ve pushed an update to your Kubernetes Deployment, and suddenly your application starts returning 500 errors. Users are complaining, and you need to restore the previous working version immediately. This is exactly the scenario where kubectl rollout undo saves the day — but you need to know how to use it safely, what happens under the hood, and how to recover from even trickier failures. This lesson gives you the complete playbook for rolling back a faulty Deployment.

The problem this lesson solves

Rolling out a broken update is one of the most common and stressful events in Kubernetes operations. A simple image tag change or config tweak can break your app in ways you didn’t test for. Without a reliable rollback strategy, you risk:

• Extended downtime while you manually rebuild old images or revert YAML files.
• Inconsistent state if you only partially revert resources (e.g., ConfigMaps but not Deployments).
• Loss of deployment history — missing the ability to see what changed and when.

The kubectl rollout undo command exists precisely to solve this: it reverts a Deployment, DaemonSet, or StatefulSet to a previous Revision in a single command, preserving your rollout history and minimising human error.

Core concept / mental model

Think of a Kubernetes Deployment as a version-controlled stack of ReplicaSets. Every time you change the pod template (image, env vars, resource limits, etc.), the Deployment creates a new ReplicaSet and gradually scales down the old one. The old ReplicaSets are not deleted — they are retained as revisions (stored as annotations on the Deployment).

Pro tip: The default retention is 10 revisions, controlled by spec.revisionHistoryLimit. Set this higher if you anticipate frequent rollbacks.

When you run kubectl rollout undo, Kubernetes: 1. Looks at the Deployment’s revision history (a list of annotated ReplicaSets). 2. Picks the previous revision (or the one you specify with --to-revision). 3. Applies the pod template from that revision to the Deployment. 4. Triggers a new rollout — scaling up the old ReplicaSet’s spec and scaling down the current one.

This is not an instant flip. The rollback is a controlled deployment with the same health checks (readiness/liveness probes) that your original rollout used. If the old revision also has bugs, the rollback can still fail — but that’s why you should test backups before you need them.

How it works step by step

Step 1: Check rollout history

Before undoing anything, see what revisions are available:

kubectl rollout history deployment/nginx-deployment

Example output:

deployment.apps/nginx-deployment
REVISION  CHANGE-CAUSE
1         kubectl apply --filename=nginx-v1.yaml --record=true
2         kubectl set image deployment/nginx-deployment nginx=nginx:1.21 --record=true
3         kubectl set image deployment/nginx-deployment nginx=nginx:1.22 --record=true

Note: The --record=true flag (or --record in older kubectl) saves the command that caused the change. In modern kubectl, use kubectl annotate or store changes in source control instead.

Step 2: Understand current state

kubectl rollout status deployment/nginx-deployment

Shows if the current rollout is complete, progressing, or failed.

Step 3: Rollback to previous revision

To roll back to the revision before the current one (revision 2 in the example above):

kubectl rollout undo deployment/nginx-deployment

Step 4: Specify a target revision

If you need to go further back, say to revision 1:

kubectl rollout undo deployment/nginx-deployment --to-revision=1

Step 5: Verify the rollback

kubectl rollout status deployment/nginx-deployment
kubectl get pods -l app=nginx

Check that pods are running and ready.

Step 6: Check the new history

After rollback, a new revision is created (revision 4) that mirrors the old template:

kubectl rollout history deployment/nginx-deployment

Now revision 4 = revision 2 in pod template, but it’s a distinct entry.

Hands-on walkthrough

Let’s simulate a faulty deployment and roll it back. You’ll need a local cluster (Minikube, Kind, or a remote cluster).

1. Create the initial Deployment

kubectl create deployment nginx-demo --image=nginx:1.20 --replicas=3 --port=80

Wait for all pods to be ready:

kubectl wait --for=condition=available deployment/nginx-demo --timeout=60s

2. Update to a faulty image

Apply a new image that doesn’t exist or has a startup error:

kubectl set image deployment/nginx-demo nginx-demo=nginx:doesnotexist

3. Observe the failure

kubectl rollout status deployment/nginx-demo

This will hang or show an error like Error: statefulset/nginx-demo rollout is failing. Check pod status:

kubectl get pods -l app=nginx-demo

You’ll see ImagePullBackOff or ErrImagePull.

4. Undo the faulty deployment

kubectl rollout undo deployment/nginx-demo

5. Confirm restoration

kubectl rollout status deployment/nginx-demo
kubectl get pods -l app=nginx-demo

Pods should revert to the image nginx:1.20 and become Running.

Full script example

# This code is for demonstration — not meant to be run directly.
# It shows the logical flow of a rollback using kubectl.

import subprocess

def deploy_and_rollback(deployment_name, namespace="default"):
    # Step 1: Deploy initial version
    subprocess.run([
        "kubectl", "set", "image", f"deployment/{deployment_name}",
        f"{deployment_name}=busybox:1.34", "--namespace", namespace
    ])

    # Step 2: Check rollout status
    result = subprocess.run([
        "kubectl", "rollout", "status", f"deployment/{deployment_name}",
        "--namespace", namespace, "--timeout=30s"
    ], capture_output=True, text=True)

    if result.returncode != 0:
        print("Rollout failed, attempting rollback...")
        # Step 3: Rollback
        subprocess.run([
            "kubectl", "rollout", "undo", f"deployment/{deployment_name}",
            "--namespace", namespace
        ])
        # Step 4: Verify rollback
        subprocess.run([
            "kubectl", "rollout", "status", f"deployment/{deployment_name}",
            "--namespace", namespace
        ])
    else:
        print("Deployment succeeded!")

Example output showing rollback

$ kubectl rollout status deployment/nginx-demo
Waiting for rollout to finish: 0 out of 3 new replicas have been updated...
Waiting for rollout to finish: 0 out of 3 new replicas have been updated...
Error: statefulset/nginx-demo rollout is failing

$ kubectl rollout undo deployment/nginx-demo
deployment.apps/nginx-demo rolled back

$ kubectl rollout status deployment/nginx-demo
deployment "nginx-demo" successfully rolled out

Compare options / when to choose what

Pro tip: kubectl rollout undo is ideal for emergency rollbacks. For planned rollbacks (e.g., after a canary failure), consider GitOps to ensure consistency with your repository.

Troubleshooting & edge cases

Common Mistakes

• Assuming --to-revision counts from 1 — revisions are sequential, but after multiple rollbacks, the current revision may not be the highest number. Always check rollout history first.
• Rolling back a ConfigMap or Secret update — rollout undo only reverts the Deployment’s pod template, not separate ConfigMaps or Secrets. If you changed a ConfigMap, you must update it separately or delete the pods to force re-read.
• Not waiting for rollout to complete — using rollout undo while a rollout is already in progress can cause inconsistent state. Always check rollout status first.
• Ignoring revisionHistoryLimit — if you’ve made many changes and the limit is 10, older revisions are gone. Increase the limit for critical deployments.

Edge Cases

• Rollback to a revision that also fails: If the old image was also broken (e.g., you upgraded from 1.20 to 1.21 and both have startup issues), rollout undo won’t help. You need to manually specify a known-good revision.
• In-flight rollback: If you run rollout undo while a rollout is progressing, Kubernetes should cancel the current rollout and start the rollback. However, this can cause temporary availability issues.
• Permissions: You need verb: create on deployments/rollback in RBAC. The edit ClusterRole includes this.

What you learned & what's next

You now understand the core mechanism of Kubernetes rollbacks: how revisions are stored, how kubectl rollout undo works step-by-step, and how to compare it with other rollback methods. You practiced rolling back a faulty deployment with hands-on commands and learned to handle common edge cases like missing revisions or failing rollbacks.

Key takeaway: Rollbacks are a safety net, not a crutch. Always test your rollback procedure before you need it.

Next up: Learn about Rolling updates and change causes — how to annotate your deployments to make rollbacks even more traceable. You’ll also explore kubectl rollout pause and resume for canary rollouts.