Secrets Rotation and Expiry

Your database password was rotated last night, or was it? If you can't answer that question with certainty, your application is carrying a ticking time bomb. Secrets that never expire are a silent liability — a leaked key in a public GitHub repo, a compromised CI token, or a database password pasted in a Slack thread becomes a permanent backdoor into your system. Implementing a robust secrets rotation and expiry strategy is no longer optional; it's a core requirement of secure software development. This lesson equips you with a practical, ordered approach to ensure your secrets are ephemeral and your systems stay resilient against credential leakage.

The problem this lesson solves

Static secrets are a gift to attackers. Once a key or password is compromised, it remains valid indefinitely, giving an intruder unlimited time and access. The consequences are real: the 2020 SolarWinds attack exploited static credentials, and the 2021 Codecov breach leaked a Docker hub token that went unnoticed for months. Manual rotation is error-prone — engineers forget, write scripts that silently fail, or skip rotation entirely because it's 'too risky'. Meanwhile, compliance frameworks like SOC 2, PCI DSS, and ISO 27001 now mandate periodic credential rotation and expiration, turning this from a best practice into a legal requirement.

This lesson solves the problem by giving you a systematic way to rotate and expire secrets automatically, without downtime and without leaving gaps. You'll learn to treat secrets as ephemeral, short-lived entities that are continuously refreshed, drastically reducing the blast radius of any single leak.

Core concept / mental model

Think of secrets like apartment keys, not house keys. A house key is permanent; once lost, you change the lock, but until then, the finder has access forever. An apartment building uses card keys that expire every 30 days. If a card is lost, it's only a temporary risk — the card stops working soon, and the building manager issues a new one. Secrets rotation is the process of replacing old credentials with new ones on a regular schedule. Expiry is setting a deadline after which a secret is no longer valid. Together, they enforce credential hygiene.

Definitions

• Rotation: The act of generating a new secret, updating consumers, and invalidating the old one.
• Expiry: A timestamp after which a secret is rejected by the system.
• Rolling window: A period during which both old and new secrets are valid, enabling zero-downtime rotation.
• Principle of least privilege applies here: a secret should only have the permissions it needs, and only for as long as needed.

Diagram-in-words: Imagine a clock face. The secret starts at 12 o'clock, valid for 24 hours. At 11:50, the system begins rotation: it creates a new secret, updates all services, and at 12:00 the old secret expires. This is the rotation window — the overlapping validity period that ensures no service is left without valid credentials.

A robust system doesn't rely on cron reminders; it uses a Secrets Manager (like AWS Secrets Manager, HashiCorp Vault, or cloud-native equivalents) that automates rotation and exposes expiry metadata to clients.

How it works step by step

Implementing secrets rotation and expiry follows a logical sequence. Here's the standard workflow:

1. Centralize secret storage: Store all secrets in a dedicated secrets manager, not in environment variables or config files. This gives you a single API to manage lifecycle.

2. Define rotation schedule: Decide on rotation frequency based on risk. For example, AWS recommends rotating database credentials every 30 days, API keys every 90 days. Use a policy that's shorter for highly privileged secrets.

3. Implement rotation lambda/function: The secrets manager triggers a function that generates a new secret, updates the target service, and stores the new secret with a future expiration date. For databases, you'd create a new user with a new password, test it, and then update the old user's password.

4. Update all consumers: Each service must fetch the latest secret from the manager at runtime, not cache it indefinitely. Use the manager's SDK to retrieve the current version.

5. Set expiration and enforce it: The secrets manager marks the old secret as expired and refuses to serve it. Clients must check the expiration field and handle expired secrets gracefully.

6. Test rotation: Practice rotating in a staging environment to ensure no service breaks. Roll back if failures occur.

7. Monitor and alert: Set up metrics on rotation success/failure, and alerts if a secret hasn't been rotated on schedule.

Cause → effect: If you skip step 4 (clients caching secrets), rotation fails in production because services still use the old secret. If you skip expiry enforcement, old secrets remain valid, defeating the purpose.

Hands-on walkthrough

Let's implement a simple rotation and expiry system using Python and the cryptography library. We'll simulate a secrets manager with an in-memory store.

Example 1: Basic Secret with Expiry

from datetime import datetime, timedelta
import secrets

class Secret:
    def __init__(self, name, value, expires_at):
        self.name = name
        self.value = value
        self.expires_at = expires_at

    def is_expired(self, now=None):
        now = now or datetime.utcnow()
        return now > self.expires_at

# Create a secret that expires in 24 hours
secret_value = secrets.token_urlsafe(32)
expiry = datetime.utcnow() + timedelta(hours=24)
secret = Secret('db_password', secret_value, expiry)

print(f"Secret created: {secret.name}")
print(f"Expires at: {secret.expires_at}")
print(f"Is expired now? {secret.is_expired()}")
# Simulate 25 hours later
later = datetime.utcnow() + timedelta(hours=25)
print(f"Is expired later? {secret.is_expired(later)}")

Output:

Secret created: db_password
Expires at: 2023-10-05 12:34:56.789012
Is expired now? False
Is expired later? True

Example 2: Automatic Rotation with Window

Now, let's implement rotation with a rolling window so consumers have time to pick up the new secret.

import time

class SecretsManager:
    def __init__(self, rotation_interval=3600, window=300):
        self.current = None
        self.previous = None
        self.rotation_interval = rotation_interval
        self.window = window
        self.next_rotation = time.time() + rotation_interval

    def rotate(self):
        # Generate new secret
        new_secret = secrets.token_urlsafe(32)
        self.previous = self.current
        self.current = new_secret
        self.next_rotation = time.time() + self.rotation_interval
        print("Rotated secrets")

    def get_secret(self, now=None):
        now = now or time.time()
        # If within rotation window, return both but mark previous as expiring
        if self.previous and (now > self.next_rotation - self.window):
            return {'current': self.current, 'previous': self.previous}
        return {'current': self.current, 'previous': None}

    def is_expired(self, secret, now=None):
        now = now or time.time()
        return secret == self.previous and now > self.next_rotation

# Simulate manager
sm = SecretsManager(rotation_interval=10)
sm.rotate()  # initial secret

# Use secret for a bit
time.sleep(2)
sm.rotate()  # rotate after 2 seconds (not t=10, but for demo)

# Check secret window
print(sm.get_secret())
time.sleep(12)
print("After expiry:", sm.is_expired(sm.previous))

Output:

Rotated secrets
{'current': '...', 'previous': '...'}
After expiry: True

Example 3: Vault-style Using hvac (HashiCorp Vault client)

For production, use a real secret manager. Here's a snippet using the hvac library to read a secret and check its lease expiration.

# pip install hvac
import hvac

client = hvac.Client(url='http://localhost:8200', token='root')

# Read a secret with a lease
read_response = client.secrets.kv.v2.read_secret_version(path='myapp/db')
secret_data = read_response['data']['data']
print("DB password:", secret_data.get('password'))

# Lease expiry is in the response for some mounts
lease_duration = read_response.get('lease_duration', 0)
if lease_duration > 0:
    print(f"Secret valid for {lease_duration} seconds")

Note: In Vault, use dynamic secrets (like database creds) which automatically expire and can be revoked.

These examples illustrate three patterns: manual expiry, rotation with overlapping windows, and leveraging an existing secrets manager. In real systems, you'll rely on the latter, but understanding the mechanics helps you debug and customize.

Compare options / when to choose what

Not all secrets are created equal, and neither are rotation strategies. Here's a comparison to guide your choice.

Pro tip: For most cloud-native applications, prefer the secrets manager. Manually rotating database passwords via cron is like using a hammer when you need a torque wrench — it works until it doesn't.

Variations to consider: - JWT rotation: For authentication tokens, implement kid (key ID) rotation in the JWT header. - OAuth client secrets: Use PKCE to avoid needing client secrets at all. - Kubernetes External Secrets operator to sync from Vault/AWS to cluster.

Troubleshooting & edge cases

1. Secret not updating despite rotation - Cause: Consumers cache the secret in memory. - Fix: Use a global variable or config manager that reloads the secret from the secrets manager on each request, or set a TTL on the cache not exceeding the rotation window.

2. Rotated secret causes database connection failures - Cause: Rotation updated the password but the database user lacks privileges to alter itself, or the new password hasn't propagated to replicas. - Fix: Test rotation in staging first. Ensure the rotation function has ALTER USER permissions. For replicas, wait for replication lag.

3. Secrets expired but application still works - Cause: The secrets manager is not enforcing expiry, or the client ignores the expiry field. - Fix: In your code, always check the expires_at attribute and treat expired secrets as invalid. Alert on any use of an expired secret.

4. Rotation fails with 'secret version not found' - Cause: The rotation function uses a token that has expired or lacks SecretsManager:PutSecretValue permission. - Fix: Use a long-lived service account for rotation, or set the rotation function's execution role explicitly.

5. Overlapping windows cause two active secrets - Cause: The old secret is still accepted during the window, leading to confusion in logs. - Fix: Implement clear naming or versioning, and ensure your log aggregation includes a version tag.

6. Secrets rotated but not invalidated - Cause: The old secret's permissions were not revoked. - Fix: After rotation, explicitly revoke or delete the old secret in the target system (e.g., drop the database user or disable the API key).

What you learned & what's next

You've mastered how to implement secrets rotation and expiry from the ground up. You understood the core problem: static secrets are a security risk, and rotation/expiry turns them into ephemeral credentials. You internalized the mental model of card keys versus house keys and the concept of rolling windows. You followed a step-by-step approach to centralize secret storage, schedule rotations, update consumers, and enforce expiry. Your hands-on examples demonstrated creating secrets with expiration timestamps, implementing a rotation manager with a window, and interfacing with a production-grade secrets manager like Vault. You can now compare the options and choose the right strategy for your scenario, and you know how to troubleshoot common pitfalls like stale caching and permission misconfigurations.

This lesson directly addressed how to explain the core idea and complete a practical exercise for secrets rotation and expiry, hitting both learning objectives. As you advance to the next lesson in the Secure development track, you'll build on this foundation to implement dynamic secrets — secrets generated on-demand with short lifetimes, further reducing exposure. Keep your secrets fresh!