Redis Memory Management: Avoiding Out-of-Memory Disasters

You're running a high-traffic Python app with Redis as your caching layer. Everything's smooth until one day, your Redis server crashes with an out-of-memory error. Your app slows to a crawl, users start seeing errors, and you're scrambling to restart services. This scenario is more common than you'd think, and it's entirely preventable.

Why Redis Runs Out of Memory

Redis stores everything in RAM by design. That's what makes it blazing fast. But RAM is expensive and finite. The problem usually creeps up slowly: you add more data, set longer TTLs, or forget to configure eviction policies. Before you know it, Redis hits its maxmemory limit and starts refusing writes.

The default behavior when Redis hits maxmemory is to return errors for write commands. That means your Python app's SET, LPUSH, or SADD calls will fail silently or throw exceptions, depending on how you handle them.

Setting Up Your Safety Net

The first thing you need to do is configure maxmemory. Without it, Redis can consume all available system RAM, which will crash not just Redis but potentially other processes on your server.

# In redis.conf
maxmemory 2gb

But setting a limit alone isn't enough. You also need an eviction policy that tells Redis what to do when memory runs out.

Choosing the Right Eviction Policy

Redis offers several eviction policies. The one you choose depends entirely on your use case.

allkeys-lru (Most Common)

This evicts the least recently used keys first, regardless of whether they have TTL set. It's the safest default for most caching scenarios.

import redis

r = redis.Redis(host='localhost', port=6379, decode_responses=True)
# Configure eviction policy at runtime
r.config_set('maxmemory-policy', 'allkeys-lru')

volatile-lru

Only evicts keys with an expiration set. If you have a mix of permanent and temporary data, this keeps your permanent data safe.

allkeys-lfu

Evicts the least frequently used keys. Good if you have a stable access pattern where some keys are rarely accessed.

noeviction

Returns errors on write operations when memory is full. This is the default, and it's dangerous for production systems.

Real-World Example: The Session Cache Disaster

At PythonSkillset, we once had a client whose Redis instance kept crashing every few weeks. They were storing user session data with 24-hour TTLs. The problem? They had 500,000 active users, each session was about 2KB, and they never set maxmemory. Redis grew until it consumed all 16GB of RAM on the server, then the OOM killer stepped in.

The fix was simple: set maxmemory to 12GB and use allkeys-lru eviction. Sessions older than a few hours got evicted first, and the system stabilized immediately.

Monitoring Memory Usage

You can't fix what you don't measure. Redis provides several ways to track memory.

Using INFO Command

import redis

r = redis.Redis()
info = r.info()
print(f"Used memory: {info['used_memory_human']}")
print(f"Max memory: {info['maxmemory_human']}")
print(f"Evicted keys: {info['evicted_keys']}")

The evicted_keys counter is your early warning system. If it's climbing, you're approaching your limit.

Memory Fragmentation

Redis memory usage isn't always straightforward. Memory fragmentation can waste 10-30% of your allocated RAM.

frag_ratio = r.info()['mem_fragmentation_ratio']
if frag_ratio > 1.5:
    print("High fragmentation detected. Consider restarting Redis.")

A fragmentation ratio above 1.5 means you're wasting significant memory. Restarting Redis during low traffic can reclaim this space.

Practical Eviction Strategies

For Caching (Most Common)

Use allkeys-lru with a generous maxmemory. This works well when your data has natural expiration through usage patterns.

import redis

r = redis.Redis()
r.config_set('maxmemory', '4gb')
r.config_set('maxmemory-policy', 'allkeys-lru')

For Session Storage

If you're storing user sessions with TTLs, volatile-ttl is your friend. It evicts keys with the shortest remaining TTL first.

r.config_set('maxmemory-policy', 'volatile-ttl')

For Rate Limiting or Counters

When you have data that must never be evicted (like rate limit counters), use noeviction but set your maxmemory high enough. Then monitor aggressively.

Monitoring That Saves You

At PythonSkillset, we've seen too many teams only check memory after a crash. Here's what you should monitor proactively.

Memory Usage Over Time

import time
import redis

r = redis.Redis()
while True:
    info = r.info()
    used = int(info['used_memory'])
    max_mem = int(info.get('maxmemory', 0))
    if max_mem > 0:
        usage_pct = (used / max_mem) * 100
        print(f"Memory usage: {usage_pct:.1f}%")
        if usage_pct > 80:
            print("WARNING: Approaching memory limit!")
    time.sleep(60)

Eviction Rate

A sudden spike in evicted keys means you're hitting your limit. This is your canary in the coal mine.

import redis
import time

r = redis.Redis()
last_evicted = 0

while True:
    current_evicted = int(r.info()['evicted_keys'])
    if current_evicted > last_evicted:
        print(f"Evicted {current_evicted - last_evicted} keys since last check")
    last_evicted = current_evicted
    time.sleep(30)

Practical Strategies to Avoid OOM

1. Set Realistic Limits

Don't set maxmemory to 100% of your available RAM. Leave at least 20% headroom for Redis's own overhead and for the operating system.

# If you have 8GB RAM, set maxmemory to 6GB
maxmemory 6gb

2. Use Appropriate Data Structures

Redis has many data types, and they use memory differently. A hash with 100 fields uses less memory than 100 separate string keys. For large datasets, consider using hashes or lists instead of individual keys.

# Bad: 1000 separate keys
for i in range(1000):
    r.set(f"user:{i}:name", f"User{i}")

# Good: One hash with 1000 fields
r.hset("users", mapping={f"user:{i}": f"User{i}" for i in range(1000)})

3. Set TTLs on Everything

Unless you have a specific reason to keep data forever, always set an expiration time. This is the single most effective memory management technique.

# Set a key that expires in 1 hour
r.setex("cached_result", 3600, "some_value")

# Or set TTL after creation
r.set("temp_data", "value")
r.expire("temp_data", 300)  # 5 minutes

4. Use Memory-Efficient Data Types

Redis has some surprising memory characteristics. A small hash with 10 fields uses about the same memory as 10 separate string keys. But a hash with 1000 fields is much more efficient than 1000 separate keys.

# Efficient: Store user data in a hash
r.hset("user:1001", mapping={
    "name": "Alice",
    "email": "alice@example.com",
    "age": 30
})

# Inefficient: Store each field as a separate key
r.set("user:1001:name", "Alice")
r.set("user:1001:email", "alice@example.com")
r.set("user:1001:age", 30)

Use Lists Sparingly

Lists in Redis are doubly linked lists. They're great for queues, but each element has overhead. If you're storing thousands of items in a list, consider using a different data structure or splitting them.

Practical Monitoring Setup

At PythonSkillset, we recommend a simple monitoring script that runs every minute.

import redis
import smtplib
from email.mime.text import MIMEText

def check_redis_memory():
    r = redis.Redis()
    info = r.info()

    used_mb = int(info['used_memory']) / 1024 / 1024
    max_mb = int(info.get('maxmemory', 0)) / 1024 / 1024

    if max_mb > 0:
        usage_pct = (used_mb / max_mb) * 100
        print(f"Redis memory: {used_mb:.0f}MB / {max_mb:.0f}MB ({usage_pct:.1f}%)")

        if usage_pct > 85:
            # Send alert
            send_alert(f"Redis memory at {usage_pct:.1f}%")

    # Check eviction rate
    evicted = int(info['evicted_keys'])
    if evicted > 1000:
        print(f"High eviction rate: {evicted} keys evicted")

Practical Tips from the Trenches

1. Set TTLs Aggressively

At PythonSkillset, we've seen teams set TTLs of 24 hours for cached API responses that change every 5 minutes. That's 24 hours of stale data eating RAM. Be realistic about how long data is useful.

# Good: Cache for 5 minutes
r.setex("api_response", 300, json.dumps(data))

# Bad: No TTL
r.set("api_response", json.dumps(data))

2. Use Redis Memory Analysis Tools

The MEMORY USAGE command tells you exactly how much memory a key consumes.

key = "user:1001"
memory_used = r.memory_usage(key)
print(f"Key '{key}' uses {memory_used} bytes")

This helps you identify bloated keys. We once found a key storing a 50MB JSON blob that was supposed to be cached for 5 minutes but had no TTL.

3. Implement a Memory Budget

At PythonSkillset, we recommend calculating your per-key memory budget. If you have 4GB of RAM and expect 1 million keys, each key can use about 4KB on average. If your data exceeds this, you need more RAM or a different strategy.

max_memory = 4 * 1024 * 1024 * 1024  # 4GB
expected_keys = 1_000_000
budget_per_key = max_memory / expected_keys
print(f"Budget per key: {budget_per_key / 1024:.1f} KB")

Handling the OOM Scenario

Even with precautions, things can go wrong. Here's what to do when Redis runs out of memory.

1. Immediate Recovery

# Restart Redis with reduced maxmemory
redis-cli CONFIG SET maxmemory 2gb
redis-cli CONFIG SET maxmemory-policy allkeys-lru

This buys you time while you investigate the root cause.

2. Identify the Culprit

Use redis-cli --bigkeys to find the largest keys. This scans your database and reports the biggest offenders.

redis-cli --bigkeys

3. Check for Memory Leaks

Sometimes the problem isn't too much data, but a memory leak in your application. Look for keys that should have expired but didn't.

# Check keys without TTL
cursor = 0
keys_without_ttl = 0
while True:
    cursor, keys = r.scan(cursor=cursor, count=1000)
    for key in keys:
        if r.ttl(key) == -1:  # No expiration
            keys_without_ttl += 1
    if cursor == 0:
        break
print(f"Keys without TTL: {keys_without_ttl}")

Advanced Techniques

Memory Sampling

Redis 7.0 introduced MEMORY STATS which gives detailed breakdowns.

stats = r.memory_stats()
print(f"Dataset size: {stats['dataset.bytes'] / 1024 / 1024:.1f} MB")
print(f"Overhead: {stats['overhead.total.bytes'] / 1024 / 1024:.1f} MB")

Using Redis as a Cache, Not a Database

This is the most common mistake. Redis is not a durable database. If you need persistence, use Redis with AOF or RDB, but understand that this adds memory overhead. For truly critical data, use PostgreSQL or MongoDB and cache in Redis.

Emergency Recovery Steps

If Redis does go OOM, here's your recovery plan.

1. Restart with Reduced Memory

redis-server --maxmemory 1gb --maxmemory-policy allkeys-lru

2. Flush Selectively

Don't run FLUSHALL unless you're desperate. Instead, delete specific key patterns.

# Delete all keys matching a pattern
cursor = 0
while True:
    cursor, keys = r.scan(cursor=cursor, match="temp:*", count=1000)
    if keys:
        r.delete(*keys)
    if cursor == 0:
        break

3. Use Redis Cluster for Horizontal Scaling

If you consistently need more than 16GB of cache, it's time to shard. Redis Cluster distributes data across multiple nodes, each with its own memory limit.

from redis.cluster import RedisCluster

rc = RedisCluster(
    startup_nodes=[
        {"host": "redis-node1", "port": 6379},
        {"host": "redis-node2", "port": 6379},
    ],
    decode_responses=True
)

The One Thing Most People Miss

Memory fragmentation. Redis allocates memory in chunks, and over time, this creates gaps. A Redis instance using 4GB of data might actually consume 6GB of RAM due to fragmentation.

The fix? Restart Redis periodically during maintenance windows. Or use MEMORY PURGE command in Redis 4.0+.

r.execute_command("MEMORY PURGE")

This doesn't delete data, but it defragments memory. Run it during low traffic.

When All Else Fails: The Emergency Plan

If Redis is already OOM and you can't restart, here's a last-resort script that deletes the largest keys.

import redis

r = redis.Redis()

def delete_largest_keys(count=100):
    # Get keys sorted by size (approximate)
    cursor = 0
    key_sizes = []
    while True:
        cursor, keys = r.scan(cursor=cursor, count=1000)
        for key in keys:
            size = r.memory_usage(key) or 0
            key_sizes.append((size, key))
        if cursor == 0:
            break

    # Sort by size descending
    key_sizes.sort(reverse=True)

    # Delete the largest keys
    for size, key in key_sizes[:count]:
        r.delete(key)
        print(f"Deleted {key} ({size} bytes)")

The Golden Rule

Never let Redis run without a maxmemory limit. It's like driving a car without a speedometer. You might be fine for a while, but eventually, you'll crash.

Start with allkeys-lru and a maxmemory that's 80% of your available RAM. Monitor your eviction rate. If you're evicting more than a few hundred keys per minute, you need more memory or a different caching strategy.

Final Thoughts

Redis memory management isn't complicated, but it requires discipline. Set limits, monitor usage, and choose the right eviction policy for your workload. Your future self (and your users) will thank you when the site stays up during traffic spikes.

Remember: Redis is a cache, not a database. Treat it like one, and you'll never see that dreaded OOM error again.