Handle Redis Failover Manually

You've built a Redis replica set, replication is humming along, and you're feeling confident about high availability. But then the primary node goes down — not gracefully, not during a maintenance window, but with a hard crash in the middle of a flash sale. Your application keeps writing to the dead primary, and your replicas are just sitting there, full of fresh data, waiting for someone to say "you're the king now." This is exactly the scenario nobody plans for and everyone freezes on. Manual failover is the emergency brake you need to pull — controlled, deliberate, and safe.

The problem this lesson solves

A Redis replica set doesn't auto-promote itself. If you're not running Redis Sentinel or Redis Cluster — or if you need to inspect state before trusting a replica — you have to perform a manual failover. The core pain: your application is pointing at a dead node, your replicas are idle, and every second of downtime costs data or revenue.

The default behavior after a primary crash is simple: replicas remain replicas. They won't vote, they won't promote. Someone — you — must decide which replica becomes the new master and tell the rest of the replicas to follow it. If you just wait, nothing happens.

Another reality: not all failovers are emergencies. You may need to move the primary role for planned maintenance — upgrading hardware, applying a kernel patch, or resharding memory. Manual failover lets you decide the when and who.

Finally, without manual failover skill, you cannot debug why an automated system (like Sentinel) made a poor choice. Understanding the manual process is the foundation of trusting (or improving) automated failover.

Core concept / mental model

Think of a Redis replica set as a band with one lead singer and several backup vocalists. The lead singer (primary) handles all the writing and most reading. The backup vocalists (replicas) just echo the lyrics (replicate data). When the lead singer loses their voice, nobody automatically steps up — the backups just keep humming along silently.

Manual failover is you, the producer, pointing at the backup vocalist on the left and shouting, "You're the lead now!" Then you tell the rest of the backups, "Follow that one."

In technical terms, manual failover involves three steps:

1. Identify the healthiest replica (closest to the primary's data, lowest replication lag).
2. Promote that replica to become the new primary (REPLICAOF NO ONE).
3. Repoint the other replicas to the new primary (REPLICAOF <new-host> <new-port>).

Crucially, you must also update your application's connection string to point to the new primary. Redis doesn't handle that part — that's on you (or a proxy like HAProxy or a client-side discovery mechanism).

Pro tip: Always check the replication offset (master_repl_offset and slave_repl_offset via INFO REPLICATION) before promoting. The replica with the smallest lag is the safest choice.

How it works step by step

Let's walk through a controlled manual failover from scratch. Assume a setup with one primary (redis-primary:6379) and two replicas (redis-replica-1:6379, redis-replica-2:6379).

Step 1: Confirm the primary is truly dead

Before acting, verify the primary is unreachable. A simple PING or INFO from a client that used to connect works.

# From app server or admin node
redis-cli -h redis-primary -p 6379 PING
# Expected: error or timeout

If the primary responds, you're dealing with a network partition, not a crash — that's a different problem (split-brain risk). For this guide, assume the primary process exited or the node is down.

Step 2: Check replication state on each replica

Connect to each replica and inspect its replication info.

redis-cli -h redis-replica-1 -p 6379 INFO replication

Look for: - role:slave — confirms it's a replica. - master_link_status:down — indicates the replica knows the primary is gone. - slave_repl_offset — the higher the number, the more data it has.

Compare offsets across replicas. Pick the one with the highest offset (i.e., the most recent data).

Step 3: Promote the chosen replica to primary

Once you've selected the best replica, promote it using the REPLICAOF command with the NO ONE argument.

redis-cli -h redis-replica-1 -p 6379 REPLICAOF NO ONE

This breaks the replica's link to the old primary and makes it a standalone primary node. Confirm with INFO replication — it should show role:master.

Step 4: Repoint the other replicas to the new primary

Now connect to each remaining replica and tell them to follow the newly promoted node.

redis-cli -h redis-replica-2 -p 6379 REPLICAOF new-primary-host 6379

Replace new-primary-host with the hostname or IP of redis-replica-1 (your new master). Verify again with INFO replication — they should now show role:slave with master_host pointing to the new primary.

Step 5: Update the application's connection string

This is outside Redis's control. Your app likely has a config file, environment variable, or service discovery mechanism. Update the Redis host/port to point to the new primary (and optionally to replicas for reads).

Blockquote callout: Do not skip Step 5. I've seen teams perform a perfect manual failover, then spend hours debugging why writes still fail — because the app still connected to the old IP.

Hands-on walkthrough

Now let's run through a full manual failover with actual commands and expected outputs. We'll simulate a three-node setup locally using Docker containers. If you don't have Docker, you can run three redis-server instances on different ports.

# Start three Redis instances on your local machine
redis-server --port 6379 --daemonize yes
redis-server --port 6380 --daemonize yes --replicaof 127.0.0.1 6379
redis-server --port 6381 --daemonize yes --replicaof 127.0.0.1 6379

Wait 2–3 seconds for replication to initialize. Confirm the replicas are connected:

redis-cli -p 6380 INFO replication
# Expected output snippet:
# role:slave
# master_host:127.0.0.1
# master_port:6379
# master_link_status:up

Step 1: Simulate primary crash

redis-cli -p 6379 DEBUG SLEEP 30 &
sleep 2
redis-cli -p 6379 PING
# Expected: Could not connect to Redis at 127.0.0.1:6379: Connection refused

We used DEBUG SLEEP to block the primary, simulating a hang/crash. (In production, you'd see a true crash.)

Step 2: Check replicas' state

redis-cli -p 6380 INFO replication
# Look for master_link_status:down
redis-cli -p 6381 INFO replication
# Same

Both replicas report the primary as down. Check offsets:

redis-cli -p 6380 INFO replication | grep slave_repl_offset
redis-cli -p 6381 INFO replication | grep slave_repl_offset

Assume 6380 has a slightly higher offset. We'll promote it.

Step 3: Promote replica-6380

redis-cli -p 6380 REPLICAOF NO ONE
# Expected: OK
redis-cli -p 6380 INFO replication
# role:master
# connected_slaves:0

Step 4: Repoint replica-6381

redis-cli -p 6381 REPLICAOF 127.0.0.1 6380
# Expected: OK
redis-cli -p 6381 INFO replication
# role:slave
# master_host:127.0.0.1
# master_port:6380
# master_link_status:up

Step 5: Verify writes on new primary

redis-cli -p 6380 SET failover_test ok
# Expected: OK
redis-cli -p 6381 GET failover_test
# Expected: "ok"

Data is intact and replicating from the new primary.

Pro tip: If your replicas were replicating from the old primary asynchronously, there may be data loss (writes that hadn't been replicated yet). Manual failover doesn't prevent that — it just minimizes downtime.

Compare options / when to choose what

Manual failover is the most basic method. Here's how it stacks up against alternatives:

When to choose manual failover: - You have a two-node setup (Sentinel needs at least three nodes). - You need to inspect data before promoting (e.g., check for corruption). - You're performing planned maintenance and want zero data loss by coordinating writes. - You're learning or debugging.

When to avoid manual failover: - You need sub-second failover. - Your team cannot respond 24/7. - You have many replicas and complex topologies.

Blockquote callout: Manual failover is not a dirty word — it's your fallback for when automation fails or needs supervision. Every senior Redis engineer knows how to do it blindfolded.

Troubleshooting & edge cases

Replica becomes primary but won't accept writes from the other replica

Problem: After promotion, the second replica fails with MASTERDOWN or cannot sync.

Fix: The most common cause is that the new primary has a different replication ID. Run REPLICAOF NO ONE again on the new primary, or restart the second replica's replication: REPLICAOF NO ONE then REPLICAOF <new-primary-host> <port>.

Data inconsistency after promotion

Problem: You promote replica A, but replica B had a higher offset and is missing writes from replica A's slightly older data.

Fix: This is asymmetry. Always verify offsets before promoting. If offsets are identical after a crash, you're safe. If not, you can use WAIT command with a timeout to sync replicas before failover, but manual failover inherently accepts this risk.

Split-brain scenario

Problem: The original primary comes back online after you promoted a new primary. Now both nodes think they're primary.

Fix: Stop the old primary immediately. Connect to it, run REPLICAOF <new-primary-host> <port> to turn it into a replica, then restart. The old primary's writes (if any) will be lost, but the system re-syncs.

Replica refuses to REPLICAOF NO ONE

Problem: REPLICAOF NO ONE returns an error about replica-serve-stale-data or similar.

Fix: Check the replica-serve-stale-data config — if set to no, the replica may reject commands during initial sync. Set it to yes (or ensure full sync before failover). Also verify you're running Redis 5.0+ (the command syntax changed slightly in 2.8–3.2).

What you learned & what's next

You now understand how to handle Redis failover manually: from verifying the primary is dead, selecting the best replica, promoting it with REPLICAOF NO ONE, and repointing other replicas. You also learned when manual failover beats automated solutions and how to troubleshoot common edge cases.

Key skills gained: - Explain the core idea behind manual failover (conscious promotion vs. automated). - Complete a practical exercise of failover with real commands. - Connect this to production scenarios like planned maintenance and emergency recovery.

Your next lesson in this track will cover Redis Sentinel configuration — a production-grade way to automate failover detection and promotion, building directly on the concepts you just practiced manually.