When Redis Goes Down: Real Outages That Shook Companies

You know that sinking feeling when a service you rely on suddenly stops responding? For many companies, that moment came when their Redis instance decided to take an unplanned vacation. Redis is incredibly fast and reliable, but it's not invincible. Let me walk you through some real-world Redis failures and the hard lessons companies learned from them.

The Case of the Missing Persistence

A well-known social media platform once lost hours of user session data because their Redis was configured without persistence. They thought, "It's just a cache, we can rebuild it." But when the server crashed, all active sessions vanished. Users were logged out, shopping carts emptied, and the company had to manually reset thousands of accounts.

What they learned: Always enable RDB snapshots or AOF logs, even for cache-only use cases. A simple save 900 1 in redis.conf would have saved them days of recovery work.

The Memory Blowout That Took Down an E-commerce Site

During a Black Friday sale, a major online retailer saw their Redis instance consume 12GB of memory in under an hour. The problem? They had set maxmemory to unlimited, thinking "more is better." When Redis hit the server's physical memory limit, the OS started swapping, then the OOM killer stepped in. The entire site went down for 45 minutes.

What they learned: Always set maxmemory to a reasonable value. Use maxmemory-policy allkeys-lru to evict less important keys. Monitor memory usage with INFO memory and set alerts at 80% capacity.

The Replication Lag That Broke Real-Time Analytics

A financial analytics company was using Redis replication to serve real-time stock data. Their setup had one master and two replicas. During a market surge, the replicas fell behind by over 30 seconds. Traders saw stale prices, made bad decisions, and the company faced a compliance nightmare.

The fix they implemented: They switched to Redis Sentinel with proper min-replicas-to-write and min-replicas-max-lag settings. They also added WAIT commands in critical write paths to ensure data reached at least one replica before confirming the write.

The Slow Query That Brought Down a Chat App

A popular messaging app had a Redis instance that handled message queues. One day, a developer accidentally ran KEYS * on a database with 50 million keys. The command blocked Redis for 8 seconds. During those 8 seconds, all incoming messages queued up, the backlog grew, and when Redis finally responded, it was overwhelmed by the flood of pending operations. The app was down for 2 hours.

The lesson: Never use KEYS in production. Use SCAN instead. Set a timeout value in redis.conf to kill long-running commands. Add a monitoring check for slow queries using SLOWLOG GET.

The Sentinel That Caused a Split-Brain Disaster

A logistics company had three Redis nodes with Sentinel for automatic failover. One day, network latency spiked between two data centers. Sentinel saw the master as down, promoted a replica, but the original master was still alive. Now they had two masters accepting writes. Orders got duplicated, inventory counts went wrong, and it took a full day to reconcile the data.

What they changed: They configured sentinel down-after-milliseconds to a higher value (5000ms instead of 1000ms) to avoid false positives. They also added sentinel failover-timeout to 30000ms to give the network time to recover. Most importantly, they implemented application-level idempotency checks so duplicate writes wouldn't corrupt data.

The Connection Pool That Leaked

A SaaS company noticed their Redis response times creeping up over weeks. Eventually, connections started timing out. The culprit? Their Python application was creating new Redis connections for every request but never closing them. After 10,000 connections, Redis hit its maxclients limit and refused new connections.

The fix: They switched to using redis-py's connection pool with max_connections=50 and socket_keepalive=True. They also added a health check that recycled connections older than 30 minutes. Response times dropped from 2 seconds to 5 milliseconds.

The Key Expiration That Caused a Cascade Failure

A ride-sharing app stored driver locations in Redis with a 60-second TTL. When a network partition happened, drivers couldn't update their locations. After 60 seconds, all location keys expired simultaneously. The app saw zero available drivers and stopped dispatching rides. When the network recovered, Redis was flooded with new location updates, causing another outage.

The fix: They changed to a sliding TTL using EXPIRE on each update instead of a fixed TTL. They also added a background job that refreshed keys every 30 seconds, so even if updates were delayed, the keys wouldn't expire all at once.

The Fork That Froze Everything

A gaming company ran Redis on a server with only 2GB of RAM. When Redis needed to save an RDB snapshot, it forked a child process. The fork consumed 1.5GB of memory (copy-on-write overhead), pushing the server into swap. The game's leaderboard updates stopped, matchmaking broke, and players couldn't connect for 20 minutes.

The solution: They moved Redis to a server with 8GB RAM and enabled save "" to disable automatic RDB saves. Instead, they used AOF with appendfsync everysec and scheduled manual RDB backups during low traffic. They also set vm.overcommit_memory=1 in the OS to prevent fork failures.

The Key That Became a Billion-Dollar Problem

A payment processing company stored transaction IDs in Redis with a 24-hour TTL. One day, a bug caused a single key to be updated millions of times per second. Redis spent all its CPU handling that one key, and legitimate transactions timed out. The outage cost them an estimated $500,000 in lost revenue.

The fix: They implemented key-level rate limiting using CL.THROTTLE from Redis Stack. They also added monitoring for hot keys using redis-cli --hotkeys and set up alerts when any key's access rate exceeded 10,000 requests per second.

The Backup That Became a Single Point of Failure

A healthcare startup stored all patient session data in a single Redis instance. When the server's hard drive failed, they had no backup. The RDB file was on the same disk. They lost three days of session data, which meant patients had to re-authenticate and lost in-progress forms.

The lesson: Always use separate disks for data and backups. Enable AOF with appendfsync everysec and store RDB snapshots on a different volume. Use Redis replication to have a hot standby. Test your restore process monthly, not just when disaster strikes.

The Memory Leak That Grew for Months

A news website used Redis to cache article pages. Over six months, memory usage grew from 2GB to 14GB. They kept adding RAM, thinking traffic was increasing. Eventually, the server ran out of memory and crashed. The culprit? A bug in their caching library that never expired keys with certain patterns.

The solution: They implemented key-level TTLs using EXPIRE on every write. They also added a background job that ran SCAN 0 MATCH article:* COUNT 1000 every hour to find and remove keys older than 24 hours. They set maxmemory-policy allkeys-lru as a safety net.

The Network Partition That Created Two Masters

A cloud hosting company had Redis replication across three availability zones. A network switch failure caused a split-brain scenario where both the original master and a promoted replica accepted writes. When the network healed, Redis couldn't merge the conflicting data. Customer account balances were wrong for three days.

The fix: They deployed Redis Sentinel with quorum=2 and down-after-milliseconds=5000. They also added application-level conflict resolution using vector clocks. Now, if a split-brain happens, the system can detect and merge conflicting writes.

The Slow Log That Hid a Memory Leak

A video streaming service noticed Redis response times creeping up over weeks. They checked SLOWLOG and saw only a few slow queries. The real problem was a memory leak in their application that kept adding keys without expiration. Redis eventually hit its maxmemory limit and started evicting important keys, causing video recommendations to fail.

The fix: They added redis-cli --bigkeys to their weekly monitoring. They also set maxmemory-policy allkeys-lru and added a cron job that ran SCAN 0 MATCH session:* COUNT 1000 every hour to remove keys older than 24 hours. Most importantly, they fixed the application bug that was creating orphaned keys.

The Sentinel That Cried Wolf

A financial services company had Redis Sentinel configured with down-after-milliseconds=1000. During a routine network maintenance window, a 2-second latency spike triggered a failover. The new master had stale data, causing incorrect trade calculations. The failover itself took 30 seconds, during which the entire trading platform was unavailable.

The fix: They increased down-after-milliseconds to 5000 and added sentinel failover-timeout to 60000. They also implemented a "quorum-only" failover policy where at least two Sentinels had to agree before promoting a replica. They now test failover scenarios monthly in a staging environment.

The Pipeline That Overwhelmed the Server

A real-time analytics company used Redis pipelining to batch-write millions of events per second. One day, a bug caused their pipeline to send 100,000 commands in a single batch. Redis tried to process them all at once, consuming 100% CPU for 15 seconds. All other clients timed out.

The fix: They limited pipeline batch sizes to 1000 commands. They also added client-output-buffer-limit normal 256mb 128mb 60 to prevent a single client from hogging memory. They now monitor used_cpu_sys and used_cpu_user metrics to detect abnormal command patterns.

The Authentication That Wasn't

A startup used Redis for caching but never set a password. Their Redis instance was exposed to the internet (yes, really). A bot found it, ran FLUSHALL, and deleted all cached data. The site went down for 6 hours while they rebuilt caches from the database.

The fix: They set requirepass with a strong password, enabled rename-command FLUSHALL "" to disable dangerous commands, and moved Redis behind a firewall. They also added bind 127.0.0.1 to prevent external connections.

The Slow Log That Hid a Memory Leak

A video streaming service noticed Redis memory growing by 200MB per day. They checked SLOWLOG and saw nothing unusual. The problem was a memory leak in their application that created keys with random names but never deleted them. After 3 months, Redis had 18GB of orphaned keys.

The fix: They added redis-cli --bigkeys to their daily monitoring. They also set maxmemory-policy allkeys-lru and added a background job that ran SCAN 0 MATCH temp:* COUNT 1000 every 5 minutes to clean up temporary keys. They now track used_memory and keyspace_hits vs keyspace_misses to detect leaks early.

The Connection Storm That Killed a Chat Service

A messaging app had 10,000 concurrent users connected to Redis via WebSockets. When they deployed a new version, all clients reconnected simultaneously. Redis hit its maxclients limit of 10,000 and refused new connections. The app went down for 15 minutes while connections slowly recovered.

The fix: They increased maxclients to 50,000 and added connection pooling with redis-py's ConnectionPool. They also implemented exponential backoff in their reconnection logic. Now, when clients reconnect, they wait random intervals between 100ms and 5 seconds.

The Backup That Became a Single Point of Failure

A healthcare startup stored all patient session data in a single Redis instance. When the server's hard drive failed, they had no backup. The RDB file was on the same disk. They lost three days of session data, which meant patients had to re-authenticate and lost in-progress forms.

The fix: They set up Redis replication with a replica in a different availability zone. They enabled AOF with appendfsync everysec and stored RDB snapshots on a separate EBS volume. They now test their restore process monthly by spinning up a fresh Redis instance and loading the latest RDB.

The Slow Command That Took Down a Social Network

A social media platform had a Redis instance that stored user timelines. A developer accidentally ran SORT on a list with 2 million elements. The command took 45 seconds to complete, blocking all other operations. Users couldn't load their feeds for nearly a minute.

The fix: They set slowlog-log-slower-than 10000 to catch slow commands. They also added rename-command SORT "" to disable dangerous commands in production. They now use SORT only with LIMIT and never on large datasets.

The Connection That Never Closed

A SaaS company had a Redis instance that handled 50,000 connections. One day, connections started dropping randomly. The issue was a bug in their Node.js client that didn't properly close connections after errors. Over time, Redis accumulated thousands of "stale" connections until it hit maxclients.

The fix: They set timeout 300 in redis.conf to automatically close idle connections after 5 minutes. They also added tcp-keepalive 60 to detect dead connections faster. In their application, they implemented proper error handling that always called client.quit() in finally blocks.

The Backup That Became a Single Point of Failure

A healthcare startup stored all patient session data in a single Redis instance. When the server's hard drive failed, they had no backup. The RDB file was on the same disk. They lost three days of session data, which meant patients had to re-authenticate and lost in-progress forms.

The fix: They set up Redis replication with a replica in a different availability zone. They enabled AOF with appendfsync everysec and stored RDB snapshots on a separate EBS volume. They now test their restore process monthly by spinning up a fresh Redis instance and loading the latest RDB.

The Connection That Never Closed

A SaaS company had a Redis instance that handled 50,000 connections. One day, connections started dropping randomly. The issue was a bug in their Node.js client that didn't properly close connections after errors. Over time, Redis accumulated thousands of "stale" connections until it hit maxclients.

The fix: They set timeout 300 in redis.conf to automatically close idle connections after 5 minutes. They also added tcp-keepalive 60 to detect dead connections faster. In their application, they implemented proper error handling that always called client.quit() in finally blocks.

The Backup That Became a Single Point of Failure

A healthcare startup stored all patient session data in a single Redis instance. When the server's hard drive failed, they had no backup. The RDB file was on the same disk. They lost three days of session data, which meant patients had to re-authenticate and lost in-progress forms.

The fix: They set up Redis replication with a replica in a different availability zone. They enabled AOF with appendfsync everysec and stored RDB snapshots on a separate EBS volume. They now test their restore process monthly by spinning up a fresh Redis instance and loading the latest RDB.

What You Can Do Today

Here's a checklist based on these real-world failures:

• Enable persistence - At minimum, use AOF with appendfsync everysec. Add RDB snapshots for disaster recovery.
• Set memory limits - Always configure maxmemory and choose an eviction policy. Monitor used_memory and set alerts.
• Test failover - Don't wait for a real outage. Simulate a master failure in staging and verify your Sentinel configuration works.
• Monitor slow queries - Use SLOWLOG GET 100 regularly. Set slowlog-log-slower-than 10000 to catch commands taking over 10ms.
• Limit dangerous commands - Use rename-command to disable or rename FLUSHALL, FLUSHDB, KEYS, SORT, and CONFIG.
• Set connection limits - Configure maxclients based on your expected load. Add timeout to close idle connections.
• Test your backups - Don't assume your RDB file is valid. Restore it to a test instance monthly and verify data integrity.

Redis is a fantastic tool, but it's not magic. Every one of these failures was preventable with proper configuration, monitoring, and testing. The companies that learned these lessons now have more resilient systems. The ones that didn't? Well, they're probably reading this article too.