Automate Failover Tests

Picture this: it's 2:47 AM on a Tuesday. Your primary PostgreSQL node dies from a kernel panic, and your failover kicks in — or does it? The pager goes off, and your team rolls out of bed to find that the standby replica failed to promote, the application pooler kept sending writes to the old primary, and your recovery.conf had an ancient typo that no one noticed for six months. You've tested failover before — manually, once, in a staging environment that looked nothing like production. But manual failover tests are like checking your fire alarm only when you smell smoke: too late, too rare, and too error-prone. This lesson shows you how to automate failover tests with chaos engineering — so the piece of infrastructure you're most afraid of breaking becomes the one you break on purpose, every week, while everyone is awake and your coffee is still hot.

The problem this lesson solves

Failover is the safety net for PostgreSQL high availability. When the primary fails, a replica is promoted to take over. But the net has holes: misconfigured replication, stale connection strings, or a health-check script that only runs on Wednesdays. The problem is that failover paths are rarely tested because shutdowns are scary and manual tests are tedious. So the first time your cluster actually fails, it fails for the first time in front of your customers. Chaos engineering fixes this by injecting failures deliberately, on a schedule, and observing the consequences. By automating these tests, you make failure a routine, repeatable event instead of an emergency experiment.

The hidden cost of manual failover tests

Manual tests have a dirty secret: they don't happen. They get postponed, skipped, or performed in a way that's so careful that nothing actually breaks. Meanwhile, the system drifts: PostgreSQL versions change, configs are tweaked, and networking rules are rewritten. When a real outage hits, every one of those unverified changes becomes a potential failure point.

Pro tip: Chaos engineering isn't about causing damage for fun. It's a scientific method: form a hypothesis ("failover will work"), break something, measure the outcome, and learn. The goal is confidence, not destruction.

Why automate?

Automation turns a one-off, error-prone process into a routine that runs on a timer. Once you automate failover tests, you get:

• Consistency — every test runs the same way, every time.
• Frequency — daily or weekly testing means the system is always in a known-good state.
• Early detection — drift in configs or code gets caught in days, not months.
• Team confidence — no more anxiety before a planned failover; you've done it a dozen times this month.

Core concept / mental model

Chaos engineering is the discipline of experimenting on a system to build confidence in its ability to withstand turbulent conditions. Failover automation is the practical application: a script or pipeline triggers a failure, monitors the recovery, and reports the results. The mental model is simple: break it on purpose, so you know it works when it breaks by accident.

A simple analogy

Think of a fire drill. You don't wait for a real fire to see if the alarms work or if everyone knows the exit route. You schedule a drill, set off the alarm, and time the evacuation. Chaos engineering is the same for your database cluster. The alarm is a simulated primary failure; the evacuation is the failover process; and the drill report is your test output showing that the replica became primary, the application reconnected, and no data was lost.

Key terms

• Primary — the PostgreSQL node that handles read/write workloads.
• Standby / Replica — a node with a hot (streaming) copy of the primary's data, ready for promotion.
• Failover — the process of promoting a standby to primary, usually automatic via a tool like patroni, repmgr, or a custom script.
• Chaos experiment — a controlled failure injection with a defined blast radius and hypothesis.
• Blast radius — the scope of the failure (e.g., one node vs. the whole cluster).

The chaos loop

Steady state → Inject failure → Observe → Compare → Improve → (repeat)

In a steady state, the system is healthy. You inject a failure (e.g., kill -9 on the primary), observe how the cluster reacts, compare the actual behavior to your hypothesis, and use the results to fix gaps. Then you repeat, because systems always drift.

How it works step by step

Automating failover tests with chaos engineering follows a repeatable sequence. Each step is designed to be safe, observable, and incremental.

Step 1 — Define the blast radius and hypothesis

Start small. If your cluster has three nodes, your first test might be stopping the primary and watching if patroni promotes a standby within a timeout. Write a clear hypothesis: "If the primary is terminated on node A, then node B will be promoted within 30 seconds, and the application will reconnect to the new primary without errors."

Step 2 — Capture the steady state

Before you break anything, record what "healthy" looks like. Use queries or metrics that answer questions like:

• Is the primary reachable?
• Is replication lag under a threshold?
• Is the application receiving no errors?

This gives you a baseline to compare against after the failure.

Step 3 — Automate failure injection

Use a script or a tool like chaostoolkit or toxiproxy to inject the failure. The failure can be:

• Process kill — simulate a crash: kill -9 on the primary postmaster.
• Network partition — block traffic between nodes, testing split-brain scenarios.
• Resource exhaustion — fill disk or memory to simulate a slow death.

Step 4 — Monitor the recovery

While the failure is happening, watch the cluster's reaction. Check promotion time, whether the old primary re-joins as a replica, and how the app's connections behave. Collect logs and metrics.

Step 5 — Compare and report

Compare the observed behavior to your hypothesis. If they match, your failover works. If not, you've found a bug. Record the results so you can track trends and detect regressions over time.

Step 6 — Schedule and repeat

Turn the test into a cron job or a CI pipeline. Run it weekly, or even daily, depending on how often you deploy changes. Any failure in the test is a failure in your production readiness.

Hands-on walkthrough

Now let's make it concrete. We'll build a minimal automated failover test for a PostgreSQL cluster managed by Patroni (a popular HA solution). You'll need two PostgreSQL nodes with Patroni running (or use a local Docker Compose setup). This example focuses on the automation logic; adjust the commands to your environment.

Setup (simplified)

Assume node1 is the primary and node2 is the standby. A script called failover_test.sh will:

1. Check cluster health.
2. Kill the primary process.
3. Wait for promotion of the standby.
4. Verify the new primary is writable.
5. Report success or failure.

Example script

#!/usr/bin/env bash
set -euo pipefail

PRIMARY_NODE="node1"
STANDBY_NODE="node2"
PATRONI_BIN="patronictl"

# Step 1: Steady state check
if ! $PATRONI_BIN -c /etc/patroni/patroni.yml list | grep -q "$PRIMARY_NODE.*paused"; then
  echo "Cluster is not in a good state" >&2
  exit 1
fi

# Step 2: Inject failure (kill the primary postmaster)
echo "$(date): Killing primary postmaster on $PRIMARY_NODE"
ssh "$PRIMARY_NODE" "pkill -9 -f \"postgres.*-D /var/lib/postgresql\""

# Step 3: Wait for promotion
for i in $(seq 1 30); do
  if $PATRONI_BIN -c /etc/patroni/patroni.yml list 2>/dev/null | grep -q "$STANDBY_NODE.*primary"; then
    echo "$(date): Promotion detected after ${i}0 seconds"
    break
  fi
  if [ "$i" -eq 30 ]; then
    echo "ERROR: Promotion did not happen within 30 seconds" >&2
    exit 1
  fi
  sleep 10
done

# Step 4: Verify new primary is writable
psql "host=$STANDBY_NODE dbname=postgres" -c "CREATE TABLE chaos_test (id int);"
if [ $? -eq 0 ]; then
  echo "FAILOVER SUCCESS: New primary is writable"
else
  echo "ERROR: New primary not writable" >&2
  exit 1
fi

echo "Failover test passed."

Expected output (on success):

Cluster is in stable state.
Killing primary on node1.
Promotion detected after 10 seconds.
FAILOVER SUCCESS: New primary is writable.
Failover test passed.

A more robust approach with a test framework

Raw shell scripts get messy as complexity grows. A better pattern is to use a dedicated chaos tool like Chaos Toolkit with a Python driver. Below is a minimal example that stops a container (simulating a node crash) and checks that the cluster recovers.

# chaos_failover_experiment.py
"""Chaos Toolkit experiment to kill a PostgreSQL primary and verify failover."""
from chaoslib.experiment import Experiment

# This is a simplified outline; real experiments use JSON/YAML
# and drivers for your fault injection (Docker, AWS, etc.)

def steady_state():
    """Return True if cluster is healthy."""
    import subprocess
    out = subprocess.check_output(["patronictl", "-c", "/etc/patroni.yml", "list"])
    # In real code parse and assert primary exists and no pause
    return b"primary" in out

def inject_failure():
    """Kill the primary node."""
    import subprocess
    subprocess.call(["ssh", "node1", "sudo", "systemctl", "stop", "postgresql"])

def verify_recovery():
    """Wait and verify new primary is writable."""
    import time
    time.sleep(15)
    # psql check
    import subprocess
    try:
        subprocess.check_call(["psql", "-h", "node2", "-c", "SELECT 1"])
        return True
    except subprocess.CalledProcessError:
        return False

# In Chaos Toolkit, these would be steps in an experiment definition
assert steady_state()
inject_failure()
assert verify_recovery(), "Recovery failed!"
print("Chaos experiment passed: failover works.")

Pro tip: Use chaostoolkit with the chaos-docker extension to test containerized clusters without touching production. Run it in a staging environment that mirrors your production topology.

Scheduling the test

Once your script passes locally, add it to cron:

# /etc/cron.d/failover-test
# Run every Sunday at 2 AM
0 2 * * 0 root /opt/scripts/failover_test.sh >> /var/log/chaos_failover.log 2>&1

Or, if you use CI/CD, trigger it via a scheduled pipeline (e.g., GitHub Actions schedule event).

Compare options / when to choose what

You can automate failover tests using several tools and approaches. Here's a comparison to help you decide:

When to choose custom scripts

If you have a single small cluster and want a 20-minute solution, a script like the one above is perfect. It's fast, transparent, and you control every line.

When to choose dedicated chaos tools

If you manage multiple clusters or need to run experiments in a CI/CD pipeline, use a tool like Chaos Toolkit or LitmusChaos. They support guardrails (e.g., automatic rollback), detailed reports, and integration with monitoring.

Variations of failure injection

Try different failure types to cover the full failure spectrum:

• Kill -9 (hard crash) — tests crash recovery and failover.
• SIGTERM (graceful stop) — tests shudown handling; it may not trigger failover if Patroni sees it as planned.
• Network timeout — use tools like tc or toxiproxy to simulate partition; tests split-brain behavior.
• Disk full — simulate storage exhaustion; tests monitoring and failover thresholds.

Troubleshooting & edge cases

Even with a solid script, chaos tests can fail for reasons other than a broken cluster. Here are common pitfalls and how to fix them:

The test kills the wrong process

Symptom: The script exits unexpectedly or affects unrelated services. Fix: Be precise. Instead of pkill -f "postgres", target the specific data directory: pkill -9 -f "/usr/lib/postgresql/14/bin/postgres.*-D /var/lib/postgresql/14/main". Verify each time with pg_stat_activity.

Failover doesn't trigger because of a misunderstanding

Symptom: The standby never promotes. Cause: Often due to replication lag exceeding the max_replication_slots or a missing promote_trigger_file. Check Patroni logs and run patronictl list before the test to ensure the cluster is healthy.

Pro tip: Before any chaos experiment, run patronictl list and ensure replication state is streaming on all standbys. A lagging replica may not be promoted.

The new primary rejects writes

Symptom: Your verification query fails with read-only transaction. Cause: The replica was promoted but still in recovery mode. Wait for the promotion to complete; sometimes it takes seconds. Also ensure hot_standby is on and the app's connection strings are updated.

Accidental data loss in a test

Symptom: Data written after the failover is missing. Cause: The original primary may have crashed before syncing the WAL. Use synchronous_commit=on and synchronous_standby_names to eliminate this risk in production. For tests, use -F to force flush, but understand the loss window.

Network partition instead of a kill

If you simulate a partition, you might get a split brain (both nodes think they are primary). Ensure your HA tool has a quorum or a tie-breaker, and always test with a reliable network isolation method.

The script leaves the cluster in a bad state

Always have a restore plan. After a test, if the old primary can't rejoin, you may need a manual pg_rewind. Automate that too in your script.

What you learned & what's next

You now understand the core idea of automating failover tests with chaos engineering: you deliberately break your PostgreSQL cluster to verify that the failover process works, and you do it on a schedule so it's never untested. You've seen a practical shell script and a taste of Chaos Toolkit, compared custom scripts versus dedicated tools, and learned how to troubleshoot common failures like lagging replicas or stuck promotions. You can now explain why chaos engineering builds confidence, and you can complete a hands-on exercise to validate your own cluster.

Next step: The next lesson in the PostgreSQL tutorial track takes you beyond failover automation and into observability — how to capture and analyze metrics like replication lag, connection errors, and query performance before and after a chaos test. You'll learn to turn chaos experiment outputs into actionable dashboards that keep your cluster resilient over time.