Automate Secret Rotation with Python

Let’s face it: manually rotating secrets in Kubernetes is a ticking time bomb. You log into the cluster, edit a Secret YAML, pray you didn’t miss a pod, and then hope the application picks up the new value on the next restart. It’s error-prone, slow, and frankly one of the last things you want to do at 2 AM during an incident. In this lesson, you’ll learn how to automate secret rotation with Python scripts — turning that fragile, manual process into a repeatable, verifiable pipeline that runs in seconds, not hours.

The problem this lesson solves

Secrets leak. Credentials expire. Rotation is not a nice-to-have — it’s a security requirement. But doing it by hand in Kubernetes is painful:

• You have to update multiple places — the Secret object, the Deployment, and sometimes the application’s config.
• Restart timing is unpredictable — even after you kubectl apply, pods keep running with the old secret until they’re restarted.
• Errors are silent — a typo in the Secret data is only discovered when your app starts failing authentication.
• Audit trails are thin — who rotated what, when, and why? Manual changes rarely answer that.

These are exactly the problems automation solves. A Python script gives you a single, idempotent path to rotate secrets — with validation, logging, and rollback — that you can run locally, in CI, or as a CronJob inside the cluster.

Core concept / mental model

Think of secret rotation like changing the locks on your house. You wouldn’t just swap the deadbolt without checking every door and window — you’d plan the sequence, verify each lock works, and keep a spare key in case something goes wrong. Your Python script is that lock-changing plan: it knows what to change, when to change it, and how to verify the change took effect.

In Kubernetes terms, the mental model is three layers:

1. The Secret object — the source of truth for credentials.
2. The workload (Deployment, StatefulSet) — the consumer of the secret.
3. The runtime state — the pods that actually hold the old value in memory.

Your script must update layer 1, force layer 3 to refresh (usually by restarting pods), and confirm layer 2 is still happy — all while keeping the system available.

How it works step by step

Here is the logical sequence a robust rotation script follows, regardless of the secret type (API key, DB password, TLS cert):

1. Fetch the current secret — read the existing data to know what you’re replacing and to keep metadata like labels.
2. Generate or retrieve the new value — from a vault, a random generator, or an external API.
3. Create the updated Secret — either by patching the existing object or creating a new one with the same name. Patching is usually safer because it preserves RBAC and references.
4. Restart the affected workloads — use kubectl rollout restart or the Kubernetes API to trigger a rolling update.
5. Wait for rollout to complete — poll the Deployment status until all replicas are updated and ready.
6. Verify the new secret works — run a health check or test endpoint that uses the secret, if available.
7. Log and clean up — record the rotation time, the old value’s fingerprint, and optionally delete the previous Secret version if you used a versioned naming scheme.

Pro tip: Always generate secrets with enough entropy. secrets.token_urlsafe(32) gives you 256 bits of randomness — far better than a hardcoded string.

Hands-on walkthrough

Let’s get practical. We’ll write two Python scripts: one that rotates a password for a Deployment, and another that does the same but as an in-cluster CronJob.

Prerequisites

Make sure you have:

• Python 3.10+
• kubectl configured to your cluster (minikube or kind for testing)
• The kubernetes and secrets packages installed:

pip install kubernetes secrets

Example 1: Rotate a password and restart a Deployment

import secrets
from kubernetes import client, config
from kubernetes.client.rest import ApiException
import time

# Load kubeconfig (from ~/.kube/config)
config.load_kubeconfig()

v1 = client.CoreV1Api()
apps_v1 = client.AppsV1Api()

NAMESPACE = "default"
SECRET_NAME = "db-password"
DEPLOYMENT_NAME = "my-api"

# 1. Fetch the current secret to preserve labels/annotations
try:
    current_secret = v1.read_namespaced_secret(SECRET_NAME, NAMESPACE)
    print(f"Current secret data keys: {list(current_secret.data.keys())}")
except ApiException as e:
    if e.status == 404:
        print("Secret not found, creating a fresh one.")
    else:
        raise

# 2. Generate a new password
new_password = secrets.token_urlsafe(24)
print(f"Generated new password (first 8 chars): {new_password[:8]}...")

# 3. Build the updated Secret body
secret_patch = {
    "data": {
        "password": v1.api_client.sanitize_for_serialization(new_password)
    }
}

# 4. Apply the patch (for existing secret) or create if missing
try:
    v1.patch_namespaced_secret(SECRET_NAME, NAMESPACE, secret_patch)
    print("Secret updated.")
except ApiException:
    # If it doesn't exist, create it
    secret_manifest = {
        "apiVersion": "v1",
        "kind": "Secret",
        "metadata": {"name": SECRET_NAME, "namespace": NAMESPACE},
        "data": {"password": v1.api_client.sanitize_for_serialization(new_password)}
    }
    v1.create_namespaced_secret(NAMESPACE, secret_manifest)
    print("Secret created.")

# 5. Restart the deployment to pick up the new secret
body = {"spec": {"template": {"metadata": {"annotations": {"kubectl.kubernetes.io/restartedAt": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())}}}}}
apps_v1.patch_namespaced_deployment(DEPLOYMENT_NAME, NAMESPACE, body)
print(f"Deployment {DEPLOYMENT_NAME} restarted.")

# 6. Wait for rollout to complete
while True:
    deployment = apps_v1.read_namespaced_deployment(DEPLOYMENT_NAME, NAMESPACE)
    if deployment.status.ready_replicas == deployment.spec.replicas:
        print("Rollout complete!")
        break
    time.sleep(2)

print("Process finished. Verify app health.")

Expected output (abridged):
Current secret data keys: ['password']
Generated new password (first 8 chars): Tc8zY...
Secret updated.
Deployment my-api restarted.
Rollout complete!
Process finished. Verify app health.

Example 2: In-cluster CronJob for scheduled rotation

To run rotation automatically every night, deploy a CronJob that runs a Python image with the same logic. Here’s a quick script that uses in-cluster config (no kubeconfig needed):

# rotate_secret_job.py
import secrets
from kubernetes import client, config
import os

config.load_incluster_config()  # Works when running inside a pod

v1 = client.CoreV1Api()
apps_v1 = client.AppsV1Api()

namespace = os.getenv("POD_NAMESPACE", "default")
secret_name = os.getenv("SECRET_NAME", "api-key")
deployment_name = os.getenv("DEPLOYMENT_NAME", "backend")

new_key = secrets.token_hex(16)

# Patch the secret
secret_patch = {"data": {"key": v1.api_client.sanitize_for_serialization(new_key)}}
v1.patch_namespaced_secret(secret_name, namespace, secret_patch)

# Restart the deployment
body = {"spec": {"template": {"metadata": {"annotations": {"rotatedAt": secrets.token_hex(4)}}}}}
apps_v1.patch_namespaced_deployment(deployment_name, namespace, body)
print("Secret rotated and deployment restarted.")

Then wrap it in a CronJob manifest:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: secret-rotator
spec:
  schedule: "0 2 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: secret-rotator-sa
          containers:
          - name: rotator
            image: python:3.12
            command: ["python", "-c", "exec(open('/scripts/rotate_secret_job.py').read())"]
            volumeMounts:
            - name: script
              mountPath: /scripts
            env:
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          restartPolicy: OnFailure
          volumes:
          - name: script
            configMap:
              name: rotation-script

Make sure the ServiceAccount has RBAC permissions to patch secrets and deployments.

Compare options / when to choose what

For a Python-driven approach, the official Kubernetes client is the sweet spot — it gives you fine-grained control without the overhead of writing raw HTTP calls.

Troubleshooting & edge cases

Here are the most common pitfalls and how to fix them:

• Pods don’t restart after Secret update — Kubernetes only restarts pods if the Secret volume is marked as changed or you explicitly trigger a rollout. Always patch the Deployment template (e.g., add a timestamp annotation) after updating the Secret.
• Secret data is base64-encoded — v1_api_client.sanitize_for_serialization() doesn’t encode it for you. You must base64-encode the value before putting it in data, or use stringData for plain text.
• In-cluster config fails — If you’re not running inside a pod, load_incluster_config() throws an exception. Fall back to load_kubeconfig() or handle the exception.
• RBAC permission denied — Even with the client, your ServiceAccount needs get, patch, and list on secrets and deployments. Use a dedicated Role and RoleBinding.
• Long-running secrets in environment variables — If your app uses env instead of mounted volumes, a Secret update won’t be picked up without a full pod recreation. Force a rolling update or switch to mounted Secrets.

Pro tip: Always test the rotation script in a non-production namespace first. Use a small script like this to verify your logic before touching real secrets.

What you learned & what's next

Congratulations! You now know how to automate secret rotation with Python scripts in Kubernetes. You’ve learned:

• The pain points of manual rotation and why automation is essential.
• A mental model of Secret → Deployment → Pod lifecycle.
• How to write a Python script using the Kubernetes client to update a Secret and restart a workload.
• How to run the same logic as an in-cluster CronJob.
• How to compare Python client vs. other options and troubleshoot common issues.

From here, you’re ready to extend this pattern to managing ConfigMaps or building a full CI/CD pipeline for secret rollout. Next in the track, we’ll dive into Managing Kubernetes Ingress with Python — but if you want to go deeper, explore adding health checks and rollback logic to your rotation script.

Keep your secrets fresh, and your deployments healthy!