Redis Sentinel Intro

You've built a Redis-backed application, and it's running well — until the server hosting your primary Redis instance goes down. Suddenly, your app can't connect to the database. Redis Sentinel solves this by automatically detecting failures, promoting a replica to new master, and reconfiguring clients — all without manual intervention. This lesson gives you a clear, hands-on introduction to Redis Sentinel so you can build production-ready, self-healing infrastructure.

The Problem This Lesson Solves

A single Redis instance is a single point of failure. If it crashes, your application is dead in the water until you manually spin up a replica or restore from backup. This downtime can cost you revenue, user trust, and sanity. Redis Sentinel addresses three critical problems:

• Automatic failure detection: Sentinel nodes constantly check if the master is alive.
• Automatic failover: If the master is unreachable, a replica is promoted to new master.
• Client notification: Sentinels tell clients who the current master is, so connections stay valid.

Without Sentinel, you'd have to implement your own health checks and manual failover scripts — error-prone and unscalable.

Pro Tip: Redis Sentinel is not a clustering solution. It provides high availability (HA), not horizontal scaling. Use Redis Cluster when you need to shard data across many nodes.

Core Concept / Mental Model

Think of Redis Sentinel as a supervisor team watching your Redis deployment. The team has three or more observers (Sentinel processes) that constantly ask the master "Are you alive?". If the master doesn't answer within a configurable timeout, a majority of Sentinels agree that the master is down, and one of them orchestrates a failover:

1. One replica is chosen as the new master.
2. The new master is configured to stop replicating.
3. Other replicas are reconfigured to replicate from the new master.
4. Clients are redirected to the new master's address.

Key Definitions

• Master: Primary Redis instance accepting writes and reads.
• Replica (formerly slave): Read-only copy that replicates from the master.
• Sentinel: Independent Redis process (port 26379) that monitors masters and replicas.
• Quorum: Minimum number of Sentinels that must agree a master is down before failover starts (typically n/2 + 1).
• Failover: Process of promoting a replica to master and reconfiguring the cluster.

The Oath of Sentinel

• Sentinels themselves must be deployed in odd numbers (3, 5, 7) to avoid split-brain scenarios.
• Each Sentinel runs as a separate daemon — not on the same port as Redis.
• Sentinel configuration is minimal: point it to your master set.

How It Works Step by Step

1. Monitoring

Every Sentinel checks the health of the master and replicas every second via PING. It also communicates with other Sentinels to share state.

2. Subjective Down (SDOWN)

If a Sentinel does not receive a valid reply from the master (or replica) within down-after-milliseconds, it marks that instance as subjectively down (SDOWN) from its own perspective.

3. Objective Down (ODOWN)

When a Sentinel decides the master is SDOWN, it asks other Sentinels: "Do you also see the master as down?" If at least quorum Sentinels agree (including itself), the master is declared objectively down (ODOWN). This triggers a failover.

4. Leader Election

The Sentinels hold an election to choose a leader to oversee the failover. The leader is chosen via the Raft consensus algorithm: the first Sentinel to gather votes from a majority becomes the leader.

5. Failover

The leader Sentinel: - Chooses the best replica (based on replication offset, priority, and run ID). - Sends SLAVEOF NO ONE to turn it into a master. - Reconfigures all remaining replicas to replicate from the new master. - Updates the internal configuration to reflect the new master.

6. Client Reconfiguration

Sentinels expose their current view of the master via SENTINEL get-master-addr-by-name <master-name>. Clients should use a Sentinel-aware driver (like redis-py with RedisSentinel) to fetch the current master address automatically.

Hands-on Walkthrough

Let's set up a minimal Sentinel deployment with one master, two replicas, and three Sentinels. We'll run everything locally using different ports.

Prerequisites

• Redis installed (redis-server, redis-sentinel)
• Python 3.10+ and redis-py installed (pip install redis)

Step 1: Start Master and Replicas

Create configuration files. Start master on port 6379, replica1 on 6380, replica2 on 6381.

# Start master (port 6379)
redis-server --port 6379 --daemonize yes

# Start replica1 (port 6380) - replicate from master
redis-server --port 6380 --slaveof 127.0.0.1 6379 --daemonize yes

# Start replica2 (port 6381) - replicate from master
redis-server --port 6381 --slaveof 127.0.0.1 6379 --daemonize yes

Verify replication:

redis-cli -p 6379 INFO replication

Step 2: Configure and Start Sentinels

Create a file sentinel1.conf:

port 26379
sentinel monitor mymaster 127.0.0.1 6379 2
sentinel down-after-milliseconds mymaster 5000
sentinel failover-timeout mymaster 10000
sentinel parallel-syncs mymaster 1

• mymaster: Logical name of the master set.
• 2: Quorum — at least 2 Sentinels (including itself) must agree master is down.
• down-after-milliseconds: Timeout for PING reply.
• failover-timeout: Max time for failover to complete.
• parallel-syncs: Number of replicas to sync simultaneously during failover.

Create sentinel2.conf (port 26380) and sentinel3.conf (port 26381) with the same content, changing only the port.

Start Sentinels:

redis-sentinel sentinel1.conf --daemonize yes
redis-sentinel sentinel2.conf --daemonize yes
redis-sentinel sentinel3.conf --daemonize yes

Step 3: Simulate Failure

Connect to master and kill it:

redis-cli -p 6379 DEBUG SLEEP 10 &
sleep 2
redis-cli -p 6379 SHUTDOWN NOSAVE

After the shutdown, Sentinel will detect the failure and promote one replica to master. Check the new master:

redis-cli -p 26379 SENTINEL get-master-addr-by-name mymaster
# Output: "127.0.0.1" "6380" (or similar)

Step 4: Connect via Python (Sentinel-aware)

from redis.sentinel import Sentinel

sentinel = Sentinel([('127.0.0.1', 26379),
                     ('127.0.0.1', 26380),
                     ('127.0.0.1', 26381)],
                    socket_timeout=0.1)

# Get the master for writes
master = sentinel.master_for('mymaster', socket_timeout=0.1)
master.set('foo', 'bar')

# Get a replica for reads (round-robin)
slave = sentinel.slave_for('mymaster', socket_timeout=0.1)
value = slave.get('foo')
print(f"Read from replica: {value}")  # Output: bar

Expected output:

Read from replica: bar

Pro Tip: Always use a sufficient socket_timeout (e.g., 0.1s) so your application doesn't hang if a Sentinel is temporarily unreachable.

Compare Options / When to Choose What

When to choose Sentinel: - You need automatic failover without sharding. - Your dataset fits on a single machine. - You want a simple, battle-tested HA solution.

When not to choose: - Your dataset exceeds RAM (use Cluster or external DB). - You need sub-second failover (consider Redis Cluster or write-behind cache patterns). - You are running only 2 Sentinels (split-brain risk).

Troubleshooting & Edge Cases

"Split-brain" — two masters become active

If the network partitions, both old and new masters might accept writes. Sentinel mitigates this by: - Requiring quorum and majority to start failover. - Reconfiguring the old master as a replica when it rejoins.

Fix: Always run an odd number of Sentinels (≥3). Monitor with SENTINEL ckquorum <master-name>.

"NOAUTH Authentication required"

If your master requires a password, configure the sentinel auth-pass directive:

sentinel auth-pass mymaster your_password_here

"Failover never starts"

Check: - Are all Sentinels reachable? Run redis-cli -p 26379 SENTINEL sentinels mymaster. - Is the quorum met? SENTINEL ckquorum mymaster. - Is down-after-milliseconds too long? Consider lowering it.

"Client still connecting to old master"

Ensure your client uses the Sentinel-based discovery (e.g., redis-py's Sentinel class). Simple redis.Redis connecting to old IP won't auto-failover.

What You Learned & What's Next

You now understand how Redis Sentinel provides automatic failover and high availability. You can: - Explain the core concepts: SDOWN, ODOWN, quorum, and leader election. - Set up a Sentinel deployment from scratch. - Connect your Python application to Redis through Sentinel-aware clients. - Troubleshoot common issues like split-brain or failover failures.

Next lesson: [Redis Cluster] — scale Redis horizontally across multiple nodes while maintaining HA.

Key Takeaways

• Redis Sentinel automates failover but does not shard data.
• Deploy Sentinels in odd numbers (≥3) to avoid split-brain.
• Client libraries like redis-py can auto-discover the current master.
• Failover takes 10–30 seconds; plan for that latency in your application.
• Test failover scenarios in staging before going to production.