Manage failover with repmgr

Your PostgreSQL primary just crashed at 2 AM. The application is down, the on-call phone is ringing, and you're scrambling to remember which standby server to promote. This is the exact pain that repmgr solves: it automates and manages failover, turning a chaotic emergency into a controlled, predictable process. In this lesson, you'll learn how to manage failover with repmgr — from the core concepts to a hands-on walkthrough that will save you when things go wrong.

The problem this lesson solves

Managing failover manually is error-prone and slow. You have a primary server and one or more standby servers, each streaming WAL data to stay in sync. When the primary dies, you need to:

1. Detect the failure.
2. Pick the most up-to-date standby.
3. Promote it to become the new primary.
4. Reconfigure all the other standbys to follow the new primary.

Doing this by hand involves multiple SSH sessions, careful pg_ctl promote commands, and a lot of finger-crossing. A single mistake—like promoting a stale standby—can cause data loss or split-brain scenarios where two servers think they're the primary. repmgr automates this entire workflow, providing monitoring, failover detection, and orchestration so you can recover in seconds, not minutes.

Core concept / mental model

Think of repmgr as the air traffic controller for your PostgreSQL cluster. It maintains a catalog of all servers in the replication group, knows their roles (primary or standby), and tracks their health and WAL position. When a failure happens, repmgr coordinates the transition, just as a controller redirects landing aircraft when a runway becomes unavailable.

The key components you'll work with:

• repmgr – the main command-line tool for cluster management.
• repmgrd – a daemon that runs on each node, watching for failures and executing failover automatically.
• repmgr.conf – the configuration file that defines each node's connection parameters and failover options.
• repmgr metadata – a set of tables in a database (usually repmgr) that store node information, including roles and last known WAL location.

Important terms to know:

• Failover – the process of promoting a standby to primary after the original primary fails.
• Promotion – making a standby writable and assigning it the primary role.
• Witness server – a lightweight node that helps the cluster determine quorum and prevent split-brain.
• Rejoin – reattaching a former primary or standby to the cluster after recovery.

How it works step by step

Setting up repmgr involves adopting a cluster, configuring each node, and then enabling the failover daemon. Here's the logical sequence:

1. Install repmgr on all nodes (primary, standbys, and optional witness).
2. Configure repmgr.conf on each node with the correct connection strings and cluster name.
3. Initialize the metadata on the primary with repmgr -f /etc/repmgr.conf primary register.
4. Clone the standby from the primary using repmgr standby clone and then register it.
5. Start repmgrd on each node to enable automatic failover detection.
6. Test failover manually with repmgr standby promote (or automatically when repmgrd detects a primary outage).

Each step builds on the previous, so a solid foundation in streaming replication is essential before you begin.

Hands-on walkthrough

Let's walk through a complete scenario where you have a primary called node1 and a standby node2. We'll see how to register the cluster, perform a manual promotion, and then rejoin the old primary.

Step 1: Configure repmgr.conf on both nodes

# /etc/repmgr.conf on node1 (primary)
node_id=1
node_name='node1'
conninfo='host=node1 dbname=repmgr user=repmgr password=secret'
log_file='/var/log/repmgr/repmgr.log'
data_directory='/var/lib/postgresql/15/main'

# /etc/repmgr.conf on node2 (standby)
node_id=2
node_name='node2'
conninfo='host=node2 dbname=repmgr user=repmgr password=secret'
log_file='/var/log/repmgr/repmgr.log'
data_directory='/var/lib/postgresql/15/main'

Step 2: Register the primary and clone the standby

On the primary (node1), run:

repmgr -f /etc/repmgr.conf primary register

Then on the standby (node2), clone and register:

repmgr -f /etc/repmgr.conf standby clone node1
repmgr -f /etc/repmgr.conf standby register

Step 3: Promote the standby manually

When the primary fails, promote node2:

repmgr -f /etc/repmgr.conf standby promote

Expected output (truncated):

NOTICE: promoting standby node2
DETAIL: promoting standby "node2" using pg_promote()
NOTICE: waiting for promotion to complete
NOTICE: STANDBY PROMOTE successful
DETAIL: node "node2" was successfully promoted to primary

Step 4: Rejoin the old primary

Once the old primary (node1) is repaired, bring it back as a standby under the new primary:

repmgr -f /etc/repmgr.conf node rejoin --force-rewind

This uses the pg_rewind mechanism to sync the old primary with the new primary and re-registers it as a standby.

Compare options / when to choose what

When to choose what:

• Manual promotion – when you need to do a planned switchover for maintenance, or when you want to verify every step.
• repmgr with repmgrd – when you want reliable automatic failover without adding a distributed consensus layer. Ideal for a simple primary/standby setup.
• Patroni – when you have multiple standby nodes and need advanced features like synchronous replication and automated leader election. Overkill for a simple two-node cluster.

For most teams starting out, repmgr with repmgrd is the sweet spot—it's lightweight, well-documented, and integrates seamlessly with streaming replication.

Troubleshooting & edge cases

"repmgr: ERROR: node 1 not in cluster"

This usually means the primary hasn't been registered, or the metadata database is not accessible. Check that the repmgr database exists and that the connection string in repmgr.conf points to the correct database.

"repmgr: ERROR: standby not registered"

If the standby was cloned but not registered, repmgr can't track it. Run repmgr -f /etc/repmgr.conf standby register on the standby after cloning.

Split-brain after network partition

If the network between the primary and standby fails, repmgrd may promote the standby while the primary is still alive, causing both to accept writes. To mitigate: - Use a witness server to establish quorum; with 2 nodes plus witness, a majority is required to call a failover. - Set failover=automatic only on the standby, and ensure the witness is reachable before promoting.

# In repmgr.conf on the witness
node_id=3
node_name='witness'
conninfo='host=witness dbname=repmgr user=repmgr password=secret'
data_directory='/var/lib/postgresql/15/main'

"pg_rewind failed" during rejoin

The old primary may have diverged too much from the new primary. This happens if the old primary had transactions that weren't replicated. Check the PostgreSQL logs and ensure wal_log_hints is enabled on both nodes to allow pg_rewind to work.

# postgresql.conf on all nodes
wal_log_hints = on

repmgrd not starting

Check that the repmgr database user has the correct permissions, and that the log file path is writable by the postgres user. Also verify that the repmgr.conf file is readable by the daemon user.

What you learned & what's next

You now understand how to manage failover with repmgr—from registering nodes and promoting a standby to rejoining a repaired primary. You've seen how repmgr automates the detection and promotion process, and you know how to avoid common pitfalls like split-brain and stale standby promotion. These skills will let you build a resilient PostgreSQL cluster with recovery times measured in seconds, not hours.

With that foundation, you're ready to move to the next lesson in the track: PostgreSQL high-availability patterns—where you'll explore strategies like synchronous replication, load balancing, and multi-data-center setups that build on the failover concepts you've just mastered.