Use Patroni for High Availability

Picture this: it’s 2 AM, your primary PostgreSQL node just died, and your app is down. Your team is scrambling to promote a standby, update connection strings, and pray the failover works. This is exactly the pain Patroni eliminates. By the end of this lesson, you’ll not only understand what Patroni is but also be able to configure a resilient cluster that survives node failures with minimal downtime. No more manual failovers — just software that does it for you, automatically and reliably.

The problem this lesson solves

Manual failover is a nightmare. When your primary PostgreSQL instance goes down, you need to:

• Promote a standby replica to become the new primary.
• Update every application connection string to point to the new host.
• Risk data loss if the old primary wasn’t fully synced.

This process is slow, error-prone, and requires a human on call. The problem is especially acute in production environments where every minute of downtime costs money and trust.

Patroni solves this by automating the entire failover process. It monitors cluster health, elects a new leader when the old one fails, and manages replication. You no longer need to manually intervene — Patroni takes care of the grunt work, so you can sleep at night (or at least worry less).

Core concept / mental model

Think of Patroni as the conductor of a PostgreSQL orchestra. Each PostgreSQL node is a musician. Patroni coordinates who plays the lead (the primary) and who follows (the replicas). When the lead suddenly drops out, Patroni selects a new lead without missing a beat.

Behind the scenes, Patroni uses a distributed consensus store — typically etcd, Consul, or ZooKeeper — to keep track of the cluster’s state. This store holds information like:

• Who is the current leader?
• What is the cluster’s configuration (e.g., replication settings)?
• Health checks and last known positions.

Every Patroni node talks to this store, so they all agree on the cluster’s state. When a node fails, the store helps the remaining nodes decide who becomes the new primary.

Key terms you’ll see

• Leader: The primary PostgreSQL node that accepts writes.
• Replica: A standby node that replicates data from the leader.
• Failover: The process of promoting a replica to become the new leader.
• Distributed consensus store: The brain that coordinates the cluster (etcd, Consul, etc.).

Why Patroni, not just streaming replication?

PostgreSQL’s built-in streaming replication is powerful, but it lacks automated failover. You still have to manually promote a standby. Patroni layers automation on top, making PostgreSQL truly highly available.

How it works step by step

Patroni’s architecture is elegant but powerful. Here’s the step-by-step flow:

1. Startup: Each node starts a PostgreSQL instance configured by Patroni. Patroni reads the cluster configuration from the distributed store and aligns the local PostgreSQL with that state.

2. Leader election: When the cluster boots, Patroni runs an election to decide the leader. The first node to register with the store typically becomes the leader, but Patroni also considers replication lag and node health.

3. Replication: The leader accepts writes and streams them to replicas using PostgreSQL’s WAL (Write-Ahead Log) streaming. Patroni ensures each replica is configured correctly.

4. Health checks: Every few seconds, each Patroni node checks the health of its local PostgreSQL and the cluster state. It reports status to the store.

5. Failover detection: If the leader fails to report health within a timeout (e.g., 30 seconds), the store’s lease expires. Replicas detect this and start a new election.

6. Promotion: A replica is promoted to become the new leader. Patroni automatically updates its configuration and begins accepting writes. The old leader, if it comes back, becomes a replica — never the primary again.

7. Recovery: Applications need to reconnect to the new leader. Patroni exposes a REST API and can also integrate with service discovery (e.g., HAProxy, Kubernetes) to automatically update endpoints.

The role of the distributed store

Without the store, Patroni couldn’t function. It provides:

• Leader lock: A key that only the leader holds. If the leader dies, the lock is released.
• Cluster state: Shared configuration and history.
• Fencing: Preventing split-brain scenarios where two nodes think they’re the leader.

Hands-on walkthrough

Let’s set up a minimal Patroni cluster with etcd on a single machine (for learning; in production you’d use separate hosts). We’ll use Docker to keep it simple, but the same config applies to VMs or bare metal.

Step 1: Install prerequisites

Make sure you have Docker and curl installed. Then create a project directory and add a docker-compose.yml:

version: '3.8'
services:
  etcd:
    image: quay.io/coreos/etcd:v3.5.4
    command: etcd --advertise-client-urls http://etcd:2379 --listen-client-urls http://0.0.0.0:2379
    ports:
      - "2379:2379"
  patroni1:
    image: patroni:latest
    environment:
      - PATRONI_ETCD_HOSTS=etcd:2379
      - PATRONI_SCOPE=mycluster
      - PATRONI_NAME=patroni1
      - PATRONI_POSTGRESQL_CONNECT_ADDRESS=patroni1:5432
      - PATRONI_RESTAPI_CONNECT_ADDRESS=patroni1:8008
      - PATRONI_POSTGRESQL_DATA_DIR=/data/pgdata
    volumes:
      - ./pgdata1:/data/pgdata
    depends_on:
      - etcd
  patroni2:
    image: patroni:latest
    environment:
      - PATRONI_ETCD_HOSTS=etcd:2379
      - PATRONI_SCOPE=mycluster
      - PATRONI_NAME=patroni2
      - PATRONI_POSTGRESQL_CONNECT_ADDRESS=patroni2:5432
      - PATRONI_RESTAPI_CONNECT_ADDRESS=patroni2:8008
      - PATRONI_POSTGRESQL_DATA_DIR=/data/pgdata
    volumes:
      - ./pgdata2:/data/pgdata
    depends_on:
      - etcd

Start the cluster:

docker-compose up -d

Step 2: Verify the cluster

Wait a few seconds, then check Patroni’s REST API on both nodes:

curl -s http://localhost:8008/patroni

You should see something like:

{
  "state": "running",
  "role": "leader",
  "cluster_unavailable": false
}

Step 3: Test failover

Now let’s kill the leader container to trigger a failover:

docker stop patroni1

Wait about 30–60 seconds, then check the other node:

curl -s http://localhost:8008/patroni | jq .

You’ll see that patroni2 is now the leader. No manual steps. That’s the magic.

Step 4: Bring the old leader back

Restart the container:

docker start patroni1

Patroni will see that patroni2 is the leader and configure patroni1 as a replica automatically.

Pro tip: In a real deployment, use a load balancer like HAProxy or a DNS record that points to the current leader via Patroni’s REST API. That way apps never connect to a stale IP.

Compare options / when to choose what

When to choose Patroni: You need high availability, want to keep your PostgreSQL on your own infrastructure, and don’t mind running a small distributed store. It’s the industry standard for self-managed HA.

Troubleshooting & edge cases

Patroni is robust, but things can go wrong. Here’s what to watch for:

1. Split-brain

Two nodes think they’re leaders. This happens when the store becomes unavailable or the health checks are misconfigured. Symptoms: Both nodes accept writes. Fix: Ensure your store is highly available, and set ttl and loop_wait appropriately. Use fencing by making sure the old leader is blocked from network access before promotion.

2. Failed prometheus or OIDC session

If the store connection drops, Patroni might get stuck. Check the logs:

docker logs patroni1 | grep ERROR

Common error: etcd cluster is unavailable or misconfigured. Verify that PATRONI_ETCD_HOSTS points to a reachable endpoint.

3. Replica lag

If a replica is far behind the leader, it can’t be promoted without losing data. Patroni monitors lag and only promotes nodes with minimal lag. Set maximum_lag_on_failover to a small value (e.g., 1MB).

4. Fencing not working

If the old leader can’t be fenced, it might try to keep writing. Use patronictl pause and manual intervention if needed.

5. Restart loops

Patroni restarting PostgreSQL repeatedly usually points to config errors. Check the JSON configuration via patronictl show-config.

Pro tip: Use patronictl switchover to gracefully move the leader during maintenance windows. It’s much safer than killing the container.

What you learned & what's next

You now understand how to use Patroni for high availability. You can:

• Explain the problem of manual failover and how Patroni solves it.
• Set up a Patroni cluster with etcd and test automatic failover.
• Compare Patroni against other approaches and know when to use it.
• Troubleshoot common edge cases like split-brain and replica lag.

Next, you’ll dive into ETF chaos testing or backup and restore strategies — but for now, practice your Patroni skills. Run a failover, study the logs, and become comfortable with the tools. High availability is not just a feature; it’s a discipline.