Why Your Redis Cache Keeps Expiring Too Early: Common Fixes

You set a TTL of 3600 seconds on your Redis key, expecting it to live for an hour. But an hour later, it's gone. Worse, sometimes it disappears in 30 minutes. If you've been scratching your head over this, you're not alone. At PythonSkillset, we've seen this issue trip up developers more often than you'd think. Let's break down why this happens and how to fix it.

The Usual Suspect: Default TTL Behavior

Redis doesn't have a built-in "default TTL" for all keys. If you don't explicitly set an expiration, the key lives forever. But here's the catch: many Redis clients and libraries have their own default TTL settings. For example, if you're using redis-py with a connection pool, the client might set a default expiration on certain operations. Check your client's documentation—sometimes the default is surprisingly short, like 300 seconds.

Fix: Always explicitly set TTL when you create a key. Use EXPIRE or SETEX commands. For example:

import redis
r = redis.Redis()
r.setex("user:123:session", 3600, "session_data")

This ensures you control the expiration, not some hidden default.

The Silent Killer: Key Eviction Policies

Redis has a memory management feature called eviction policies. When your Redis instance runs low on memory, it starts evicting keys—even if they haven't expired yet. The default policy is noeviction, which means Redis will return errors on writes when memory is full. But many production setups use policies like allkeys-lru or volatile-ttl. These can delete keys before their TTL is up, especially if you're storing a lot of data.

Real-world example: At PythonSkillset, we once had a caching layer for user profiles. The cache was set to expire in 24 hours, but profiles kept disappearing after 2 hours. The culprit? The allkeys-lru eviction policy was kicking in because the Redis instance was running at 90% memory usage. The fix was to either increase memory or switch to volatile-lru (which only evicts keys with TTL set) and monitor memory usage.

Fix: Check your Redis config with CONFIG GET maxmemory-policy. If it's allkeys-lru or allkeys-random, consider switching to volatile-lru or volatile-ttl if you only want to evict keys with TTLs. Also, monitor memory usage with INFO memory and set maxmemory appropriately.

The TTL Reset Trap

This one is sneaky. You set a TTL on a key, but later you update the key's value using SET without specifying a new TTL. In Redis, SET without EX or PX removes any existing TTL. So if you do:

r.setex("cache:data", 3600, "old_value")
# Later...
r.set("cache:data", "new_value")  # This removes the TTL!

Now the key lives forever, or until memory pressure evicts it. But if you're using a pattern like "set with TTL, then update," you might accidentally reset the expiration to "never."

Fix: Always use SET with EX or PX if you want to preserve or update the TTL. Or use EXPIRE after the update:

r.set("cache:data", "new_value")
r.expire("cache:data", 3600)

The Clock Drift Problem

Redis uses the server's system clock to track TTLs. If your server's clock is drifting—say, due to NTP issues or a misconfigured VM—the TTL calculations can be off. This is rare but happens in cloud environments with clock skew.

Fix: Ensure your Redis server has NTP enabled and the clock is synchronized. You can check with date on the server. If you're using a managed Redis service (like AWS ElastiCache or Redis Cloud), this is usually handled for you.

The "EXPIRE" Command on Non-Existent Keys

This is a classic rookie mistake. You call EXPIRE on a key that doesn't exist yet. Redis returns 0 (failure), and no TTL is set. Later, when you set the key, it has no expiration.

Example:

r.expire("temp:data", 300)  # Key doesn't exist yet, returns 0
r.set("temp:data", "value")  # Now key exists, but no TTL

Fix: Always set the TTL at the same time as the key. Use SETEX or combine SET with EXPIRE in a pipeline.

The "EXPIRE" on a Key That Already Has a TTL

This one is counterintuitive. If you call EXPIRE on a key that already has a TTL, it replaces the old TTL with the new one. So if you have a background job that refreshes the TTL every 5 minutes, but the original TTL was 1 hour, you're actually resetting the clock every 5 minutes. That's fine if that's what you want. But if you accidentally call EXPIRE with a shorter time, you'll shorten the key's life.

Real-world example: At PythonSkillset, we had a caching system that refreshed user sessions every 10 minutes. The original TTL was 1 hour. But a bug in the refresh logic called EXPIRE with 300 seconds (5 minutes) instead of 3600. Sessions were expiring in 5 minutes, not 1 hour. The fix was to use TTL to check the remaining time before refreshing.

Fix: Use TTL to check remaining time before resetting. Or use GETEX (Redis 6.2+) to get the value and set a new TTL atomically.

The "EXPIRE" on a Key That's Already Expired

This sounds obvious, but it's easy to miss. If you call EXPIRE on a key that has already expired (TTL = -2), Redis returns 0 and does nothing. But if you call EXPIRE on a key that has no TTL (TTL = -1), it sets the TTL. The problem is when you have a race condition: two processes try to set the same key, one sets a TTL, the other overwrites it without a TTL.

Fix: Use atomic operations like SETNX or Lua scripts to ensure TTL is set consistently. Or use SET with NX and EX options:

r.set("cache:data", "value", ex=3600, nx=True)

The "PERSIST" Command

This one is rare but worth mentioning. If you or your code calls PERSIST on a key, it removes the TTL entirely. This can happen accidentally in complex codebases where multiple services interact with the same Redis instance.

Fix: Audit your code for PERSIST calls. Use TTL to check before calling PERSIST if you're unsure.

The "RENAME" Command

When you rename a key, the TTL is preserved. But if you rename a key to an existing key, the old key's TTL is lost. This is documented behavior, but it's easy to forget.

Example:

r.setex("temp:old", 3600, "data")
r.rename("temp:old", "temp:new")  # TTL preserved
# But if "temp:new" already exists, its TTL is overwritten

Fix: Be careful with RENAME. Use RENAMENX if you want to avoid overwriting existing keys.

The "FLUSHALL" or "FLUSHDB" Mistake

This one is obvious but worth mentioning. If you or a script accidentally runs FLUSHALL or FLUSHDB, all keys are deleted immediately, regardless of TTL. This is a common cause of "early expiration" in development environments.

Fix: Use FLUSHDB only in development. In production, use FLUSHALL ASYNC to avoid blocking. And never run these commands in production without a backup.

The "EXPIREAT" with Unix Timestamp Confusion

EXPIREAT sets a key to expire at a specific Unix timestamp. If you pass a timestamp in the past, the key is deleted immediately. This is a common bug when you calculate the timestamp incorrectly.

Example:

import time
r.expireat("cache:data", int(time.time()) - 3600)  # Expired an hour ago!

Fix: Use EXPIRE with seconds from now instead of EXPIREAT unless you really need absolute time. Or double-check your timestamp calculations.

The "TTL" Command Returns -1 or -2

If you check TTL and get -1, the key has no expiration. If you get -2, the key doesn't exist. This can confuse debugging. You might think the key expired early, but it was never set with a TTL in the first place.

Fix: Always verify with TTL after setting a key. Use EXISTS to check if the key is still there.

The "Redis Cluster" Gotcha

In Redis Cluster, keys are distributed across nodes based on hash slots. If you set a TTL on a key, it's stored on the node that owns that key's hash slot. But if you later access the key from a different node (due to a client misconfiguration), you might get a redirect error or stale data. This doesn't cause early expiration per se, but it can make it seem like the key expired because you're looking at the wrong node.

Fix: Use a Redis client that handles cluster routing correctly, like redis-py-cluster. Always connect to the cluster with the right configuration.

The "EXPIRE" on a Key with a Very Short TTL

This is a logic error. You set a TTL of 1 second, then wonder why it's gone after 2 seconds. But sometimes it's more subtle: you set a TTL of 3600 seconds, but your code has a bug that resets it to 60 seconds elsewhere.

Fix: Audit your code for all places where TTL is set. Use a consistent pattern, like a wrapper function that always sets TTL the same way.

The "Redis Timeout" Misconception

Redis has a timeout configuration that closes idle connections. This doesn't affect key expiration, but it can cause your client to think the key expired because it can't connect. If your client has a short connection timeout, it might throw an error that looks like the key is gone.

Fix: Increase the client's socket timeout. In redis-py, set socket_timeout and socket_connect_timeout to reasonable values (e.g., 5 seconds).

The "Lazy Expiration" Myth

Redis doesn't actively scan all keys to check expiration. Instead, it uses a lazy expiration strategy: when a key is accessed, Redis checks if it's expired and deletes it. It also has a background process that samples a few keys every 100ms and deletes expired ones. This means a key can technically live a few seconds past its TTL, but not much. If you're seeing keys disappear before their TTL, it's not lazy expiration—it's one of the above issues.

The "Memory Pressure" Factor

Even with volatile-lru policy, if Redis is under severe memory pressure, it might evict keys with TTLs before they expire. This is by design: Redis prioritizes keeping the most recently used keys. If your cache is too large for the available memory, old keys get evicted.

Fix: Monitor memory with INFO memory and set maxmemory to a reasonable value. Use MEMORY USAGE to check how much memory a key uses. Consider using a smaller key size or compressing values.

The "Key Name Collision" Issue

If two different parts of your application use the same key name, one might set a TTL and the other might overwrite it without a TTL. This is common in microservices architectures.

Fix: Use namespacing for keys, like app:module:key. Document key naming conventions across your team.

The "Redis Version" Quirk

Older versions of Redis (before 2.1.3) had a bug where EXPIRE on a key that already had a TTL would reset the TTL incorrectly. This is ancient history now, but if you're running an old Redis instance, upgrade.

The "Client Library" Bug

Some Redis client libraries have bugs with TTL handling. For example, older versions of redis-py had a bug where SETEX with a TTL of 0 would create a key that never expires. Always use the latest stable version of your client library.

The "Network Latency" Factor

If your application sets a TTL of 3600 seconds, but the network round trip takes 100ms, the key will actually expire 3600.1 seconds after the command was sent. This is negligible for most use cases, but if you're setting very short TTLs (like 1 second), network latency can cause early expiration.

Fix: Add a small buffer to your TTL. For example, if you need a key to live for at least 1 second, set TTL to 2 seconds.

The "Multiple Clients" Problem

If you have multiple processes or services writing to the same Redis instance, one might set a key with a TTL, and another might overwrite it without a TTL. This is common in microservices architectures.

Fix: Use a centralized caching service or a Redis proxy that enforces TTL policies. Or use Lua scripts to atomically set keys with TTL.

The "Redis Sentinel" Failover Issue

In a Redis Sentinel setup, if the master fails and a replica is promoted, the new master might have slightly different data. TTLs are replicated, but if the failover happens during a write, some keys might be lost. This is rare but can cause "early expiration" in the sense that keys disappear.

Fix: Use Redis Cluster instead of Sentinel for better consistency. Or use a managed Redis service that handles failover transparently.

The "Memory Fragmentation" Factor

Redis uses memory allocators like jemalloc. Over time, memory can become fragmented, causing Redis to use more memory than expected. This can trigger eviction policies even if your data size seems small.

Fix: Monitor memory fragmentation with INFO memory and look at mem_fragmentation_ratio. If it's above 1.5, consider restarting Redis or using MEMORY PURGE (Redis 4.0+).

The "Key Pattern" Issue

If you're using Redis as a cache with a pattern like cache:user:{id}:profile, and you set TTL on each key individually, but you also have a background job that deletes keys matching a pattern (e.g., cache:user:*), that job might delete keys before their TTL.

Fix: Use SCAN with a pattern to delete keys, but be careful not to delete keys that are still valid. Or use Redis's built-in key expiration instead of manual deletion.

The "Time Zone" Confusion

Redis uses Unix timestamps internally, which are timezone-agnostic. But if your application calculates TTL based on local time and then converts to UTC incorrectly, you might set a TTL that's shorter than intended.

Example: You want a key to expire at midnight local time. You calculate the seconds until midnight in your local timezone, but Redis stores it as UTC. If your local timezone is UTC+5, the key will expire 5 hours early.

Fix: Always work in UTC when calculating TTLs. Use datetime.utcnow() in Python.

The "Key Expiration and Re-creation" Pattern

This is a common pattern: you set a key with a TTL, then later you check if the key exists. If it doesn't, you recreate it. But if the key is recreated with a new TTL, the old TTL is gone. This is fine, but if you're not careful, you might recreate the key with a shorter TTL.

Fix: Use SET with NX (set only if not exists) to avoid overwriting valid keys.

The "Redis Transaction" Issue

In a Redis transaction (MULTI/EXEC), commands are queued and executed atomically. But if you set a TTL inside a transaction, and another client modifies the key between the transaction's execution, the TTL might be lost. This is rare but possible.

Fix: Use Lua scripts for atomic operations that need to set TTL and value together.

The "Key Expiration and Replication" Issue

In Redis replication, TTLs are replicated to replicas. But if a replica is promoted to master, the TTLs are preserved. However, if you have a replica that's lagging behind, it might have stale TTLs. This doesn't cause early expiration, but it can cause confusion.

Fix: Use Redis Sentinel or Cluster for high availability. Monitor replication lag with INFO replication.

The "Memory Overhead" Factor

Redis stores keys and values with some overhead. If you have many keys with short TTLs, the memory overhead can be significant. This can trigger eviction policies even if your data seems small.

Fix: Use hash data structures to store multiple fields under one key. For example, instead of user:1:name, user:1:email, use user:1 as a hash with fields name and email. This reduces key count and memory overhead.

The "Key Expiration and Persistence" Issue

If you're using Redis persistence (RDB or AOF), expired keys are not written to disk. But when Redis restarts, it loads the RDB or AOF file, and keys that were expired at the time of the snapshot are gone. This can cause "early expiration" after a restart.

Fix: This is expected behavior. If you need keys to survive restarts, don't set TTLs, or use a database instead of Redis for persistent data.

The "Client-Side Caching" Confusion

Some Redis clients have client-side caching (e.g., Redis 6's client-side caching). If your client caches a key locally, it might serve stale data even after the key expires in Redis. This can make it seem like the key expired early because you're seeing old data.

Fix: Disable client-side caching if you don't need it. Or use CLIENT TRACKING with proper invalidation.

The "Network Partition" Issue

In a distributed system, a network partition can cause your application to lose connection to Redis. When the connection is restored, the key might have expired in the meantime. This is not a Redis bug, but it can feel like early expiration.

Fix: Use connection retry logic with exponential backoff. Monitor network latency.

The "Key Expiration and Lua Scripts" Issue

If you use Lua scripts that manipulate keys, be aware that Redis's Lua sandbox doesn't have access to the current time. You have to pass the TTL as an argument. If you hardcode a TTL in the script, it might be wrong.

Fix: Always pass TTL as an argument to Lua scripts.

The "Redis 7.0" New Feature: EXPIRE with NX/XX/GT/LT

Redis 7.0 introduced new options for EXPIRE: NX (set only if no TTL), XX (set only if TTL exists), GT (set only if new TTL is greater), and LT (set only if new TTL is less). These can help avoid accidental TTL overwrites.

Example:

r.expire("cache:data", 3600, nx=True)  # Only set TTL if none exists

The "Application Logic" Bug

Sometimes the problem isn't Redis at all—it's your application. You might have a bug where you delete the key manually, or you have a background job that cleans up old keys but uses the wrong criteria.

Fix: Add logging around key creation and deletion. Use Redis's MONITOR command to see all commands in real-time (careful in production).

The "Time-to-Live vs. Time-to-Idle" Confusion

Redis has two expiration mechanisms: TTL (time to live) and IDLETIME (time since last access). OBJECT IDLETIME shows how long a key has been idle, but it doesn't affect expiration. Some developers confuse this with TTL and think the key should expire after being idle.

Fix: Use EXPIRE for absolute expiration. If you need idle-based expiration, you'll have to implement it in your application logic.

The "Redis 6.0" Client Tracking Issue

Redis 6.0 introduced client tracking for cache invalidation. If you're using this feature, keys can be invalidated on the client side before they expire in Redis. This can make it seem like the key expired early.

Fix: Understand how client tracking works. It's designed for server-assisted client-side caching, not for key expiration.

The "Key Expiration and Replication Lag" Issue

In a master-replica setup, TTLs are replicated as absolute Unix timestamps. If the replica's clock is slightly behind the master's, the key might expire a few seconds later on the replica. This doesn't cause early expiration, but it can cause inconsistency.

Fix: Ensure all Redis nodes have synchronized clocks via NTP.

The "EXPIRE" on a Key with a Negative TTL

If you pass a negative TTL to EXPIRE, Redis treats it as an immediate deletion. This is a common bug when you calculate TTL as current_time - future_time instead of future_time - current_time.

Fix: Always ensure your TTL calculation yields a positive number. Use max(0, ttl) as a safety check.

The "Redis 7.0" New Feature: EXPIRE with NX/XX/GT/LT

Redis 7.0 introduced these options to give you more control. For example, EXPIRE key seconds NX only sets the TTL if the key doesn't already have one. This can prevent accidental overwrites.

Example:

r.expire("cache:data", 3600, nx=True)  # Only set if no TTL exists

The "Key Expiration and Replication" in Redis Cluster

In Redis Cluster, TTLs are replicated to replicas. But if