Redis Persistence Explained: RDB vs AOF for Data Safety

If you're using Redis in production, you've probably wondered what happens to your data when the server restarts. Unlike traditional databases that write everything to disk immediately, Redis keeps most of its data in memory for speed. That's great for performance, but it means you need a solid plan for data safety. That's where persistence comes in.

Redis offers two main persistence options: RDB (Redis Database) snapshots and AOF (Append-Only File). Each has its own strengths and weaknesses, and choosing the right one depends on your specific needs. Let's break them down.

RDB Snapshots: The Quick and Efficient Option

RDB works by taking periodic snapshots of your entire dataset and saving them to disk. Think of it like taking a photograph of your data at a specific moment. When Redis restarts, it loads the most recent snapshot back into memory.

How it works: You configure a save interval, like "save 900 1" (save every 15 minutes if at least one key changed). Redis forks a child process that writes the entire dataset to a temporary file, then replaces the old snapshot. This is efficient because the child process handles the disk I/O while the parent continues serving requests.

The good parts: - Very fast recovery times. Loading a single RDB file is much quicker than replaying an AOF log. - Compact file size. RDB files are compressed, making them ideal for backups or transferring between servers. - Minimal performance impact during normal operation. The fork process uses copy-on-write, so the main Redis process isn't blocked.

The trade-offs: - You can lose data between snapshots. If Redis crashes 10 minutes after the last save, those 10 minutes of writes are gone. - Forking can be resource-intensive for large datasets. If your Redis instance has 50GB of data, the fork might cause a noticeable pause.

AOF: The Granular Approach

AOF takes a different approach. Instead of periodic snapshots, it logs every write operation to an append-only file. Think of it as a detailed diary of every change made to your data.

How it works: Every write command (SET, INCR, LPUSH, etc.) gets appended to the AOF file. You can configure how often Redis syncs this file to disk: every second, every write, or never. The "every second" option is the most common balance between safety and performance.

The good parts: - Much finer granularity. With "appendfsync everysec," you'll lose at most one second of data if Redis crashes. - The AOF log is human-readable and can be edited manually if needed (though this is rarely a good idea). - Redis can rewrite the AOF file in the background to keep it from growing indefinitely.

The trade-offs: - AOF files are typically larger than RDB files for the same dataset. - Recovery can be slower because Redis must replay every command from the log. - The AOF format is more complex, which means there's a slightly higher chance of corruption (though Redis has tools to fix this).

Which One Should You Use?

The answer depends on your application's needs. Let me give you some real-world scenarios.

Use RDB if: - You're using Redis primarily as a cache where losing a few minutes of data is acceptable. - You need fast restarts after maintenance or upgrades. - You want to take daily backups to S3 or another storage service. RDB files are perfect for this because they're compact and self-contained.

Use AOF if: - You can't afford to lose more than a second of data. Financial systems, real-time analytics, or any application where every write matters. - You need a more durable audit trail of all operations. - Your dataset is small enough that AOF recovery time isn't a concern.

Use both if: - You want the best of both worlds. Redis supports running RDB and AOF simultaneously. In this case, Redis uses the AOF for recovery (since it's more complete) but still generates RDB snapshots for backups.

Real-World Example: PythonSkillset's Setup

At PythonSkillset, we run a mix of both depending on the use case. For our session cache, we use RDB with a 5-minute save interval. Losing a few minutes of cached sessions isn't catastrophic, and the fast recovery time means our users don't experience delays after a restart.

For our analytics queue, we use AOF with "appendfsync everysec." We're tracking user interactions and page views, and losing even a few seconds of that data would skew our reports. The slightly slower recovery time is a trade-off we're willing to make for data integrity.

Practical Configuration Tips

Here's what I've learned from running Redis in production:

For RDB:

# Save every 5 minutes if at least 100 keys changed
save 300 100
# Save every hour if at least 1000 keys changed
save 3600 1000
# Disable RDB if you're only using AOF
save ""

For AOF:

appendonly yes
appendfsync everysec  # Good balance
# appendfsync always  # Maximum safety, slower writes
# appendfsync no      # Let OS decide, riskier

Pro tip: If you're using AOF, enable the auto-rewrite feature. It prevents the AOF file from growing indefinitely:

auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb

The Hybrid Approach

Since Redis 4.0, there's a hybrid mode that combines both. When enabled, Redis uses an RDB snapshot as the base and appends incremental AOF changes on top. This gives you the fast recovery of RDB with the durability of AOF. It's enabled by default in Redis 5 and later.

Common Mistakes to Avoid

1. Running without persistence in production. I've seen teams lose hours of data because they assumed Redis would never restart. Always configure at least one persistence method.

2. Using "appendfsync always" without understanding the cost. This option syncs every write to disk, which can reduce throughput by 50% or more. Only use it if you truly need that level of durability.

3. Ignoring disk space. AOF files can grow quickly under heavy write loads. Monitor your disk usage and set up alerts.

4. Forgetting about backup rotation. Old RDB files and AOF logs can eat up disk space. Set up a retention policy.

Testing Your Setup

Don't just configure persistence and forget about it. Test it. Here's a simple way to verify your setup works:

1. Start Redis with your persistence configuration.
2. Write some test data.
3. Kill the Redis process (simulate a crash).
4. Restart Redis and check if your data is there.

You'd be surprised how many people skip this step and discover their persistence isn't working when they actually need it.

The Bottom Line

Redis persistence isn't complicated, but it requires thought. RDB is your go-to for speed and simplicity. AOF is your choice when every write matters. And using both gives you the most robust setup.

The key is understanding your application's tolerance for data loss. If you're building a caching layer, RDB is probably fine. If you're storing user sessions or transaction data, lean toward AOF. And if you're not sure, start with both and monitor how it performs.

Remember, Redis is fast because it works in memory. Persistence is the safety net that makes it production-ready. Don't skip it.