Cache Queries with Redis

Imagine your application making the same expensive database query hundreds of times per second — each call hitting the disk, parsing SQL, building rows, and sending data over the network. The result? Sluggish response times, a strained database server, and unhappy users. Caching database queries with Redis offers a simple, lightning-fast escape: store the result of that query once, then serve it from Redis's in-memory key-value store for subsequent requests. This lesson shows you exactly how to implement query caching with Redis, turning repeated reads into near-instant responses.

The problem this lesson solves

Database queries are often the slowest part of an application. A typical SELECT that joins multiple tables can take tens or hundreds of milliseconds — an eternity in the world of HTTP APIs. When the same data is requested by many users (or the same user refreshes), you pay that cost over and over. This leads to:

• Increased database load — the DB must work harder, slowing down writes and other critical queries.
• Higher latency — end users feel the delay, leading to poor UX and lost revenue.
• Scalability bottlenecks — as traffic grows, the database becomes the first component to fail.

By caching the query result in Redis, you reduce the number of round trips to the database. The first request fetches data from the DB and stores it in Redis with a time-to-live (TTL). Subsequent requests read from Redis — often in under a millisecond. Your database breathes a sigh of relief, and your users enjoy snappy responses.

Core concept / mental model

Think of Redis as a super-fast scratchpad sitting right in front of your database. When a query comes in:

1. Check the scratchpad — Ask Redis if the result is already stored.
2. Cache hit — If found (and not expired), return immediately.
3. Cache miss — If not found, run the query on the database, store the result in Redis with a TTL, then return.

Cache-aside pattern

The most common pattern for query caching is cache-aside (also called lazy loading). The application is responsible for both populating and invalidating the cache. This keeps your database as the source of truth — the cache is just a performance accelerator.

Key concepts

• Cache key: A unique string that identifies the query (e.g., "user:profile:42" or "posts:recent:10").
• Cache value: The serialized result — often JSON, a pickled Python object, or a simple string.
• TTL (Time-To-Live): How long the cache entry lives before Redis automatically evicts it. Choose a TTL that matches how often your data changes (e.g., 60 seconds for a blog post list, 300 seconds for a user profile).
• Cache invalidation: When data in the database changes, you must delete or update the corresponding cache key to avoid serving stale data.

Pro tip: Never hardcode TTLs. Use configuration or constants, and always set a TTL — otherwise stale data lives forever.

How it works step by step

Let's trace a typical cache-aside workflow for a Python web app that uses Redis and a SQL database (like PostgreSQL).

Step 1: Build a unique cache key

The key must uniquely identify the query and its parameters. A good practice is to include: - The entity name (e.g., user) - The action or query type (e.g., profile) - Any identifiers (e.g., user_id)

Example: "user:profile:42"

Step 2: Check Redis first

Use redis.get(key) to attempt a read. If it returns a non-None value, you have a cache hit. Deserialize it and return.

Step 3: On cache miss, query the database

If the key is missing (or expired), run the database query. This is the slow path — but it happens only once per TTL window.

Step 4: Store the result in Redis

Use redis.setex(key, ttl, value) to store the serialized result with an expiration. The setex command is atomic — set value and TTL in one operation.

Step 5: Return the result

Whether from cache or database, return the data to the caller.

Step 6: Invalidate on updates

When the underlying data changes (e.g., user updates their profile), delete the corresponding cache key using redis.delete(key). On the next read, the cache will miss, and fresh data will be stored.

Hands-on walkthrough

Let's implement a simple cache-aside pattern for a PostgreSQL query. We'll use redis-py and psycopg2. Make sure Redis is running locally (default port 6379).

Setup

First, install the required packages:

pip install redis psycopg2-binary

Example 1: Basic caching of a user profile

import json
import redis
import psycopg2
from psycopg2.extras import RealDictCursor

# Redis connection
redis_client = redis.Redis(host='localhost', port=6379, db=0)

# Database connection (simplified — use connection pooling in production)
def get_db_connection():
    return psycopg2.connect(
        host='localhost',
        dbname='mydb',
        user='myuser',
        password='mypass'
    )

def get_user_profile(user_id: int) -> dict:
    cache_key = f"user:profile:{user_id}"
    # Step 2: Check Redis
    cached = redis_client.get(cache_key)
    if cached is not None:
        print("Cache HIT")
        return json.loads(cached)
    # Step 3: Cache miss — query DB
    print("Cache MISS — fetching from DB")
    conn = get_db_connection()
    try:
        with conn.cursor(cursor_factory=RealDictCursor) as cur:
            cur.execute("SELECT id, name, email FROM users WHERE id = %s", (user_id,))
            row = cur.fetchone()
    finally:
        conn.close()
    # Step 4: Store in Redis with TTL of 300 seconds
    redis_client.setex(cache_key, 300, json.dumps(row))
    return row

# Usage
print(get_user_profile(42))
print(get_user_profile(42))  # Second call hits cache

Expected output (first run):

Cache MISS — fetching from DB
{'id': 42, 'name': 'Alice', 'email': 'alice@example.com'}
Cache HIT
{'id': 42, 'name': 'Alice', 'email': 'alice@example.com'}

Example 2: Caching a list query with parameters

import json
import hashlib

def get_recent_posts(limit: int = 10) -> list:
    # Build a deterministic key from the query parameters
    params_str = f"posts:recent:limit={limit}"
    cache_key = f"posts:recent:{hashlib.md5(params_str.encode()).hexdigest()}"

    cached = redis_client.get(cache_key)
    if cached:
        print("Cache HIT")
        return json.loads(cached)

    print("Cache MISS — fetching from DB")
    conn = get_db_connection()
    try:
        with conn.cursor(cursor_factory=RealDictCursor) as cur:
            cur.execute("SELECT id, title, created_at FROM posts ORDER BY created_at DESC LIMIT %s", (limit,))
            rows = cur.fetchall()
    finally:
        conn.close()
    redis_client.setex(cache_key, 60, json.dumps(rows))
    return rows

print(get_recent_posts(5))

Example 3: Cache invalidation on update

def update_user_name(user_id: int, new_name: str) -> None:
    conn = get_db_connection()
    try:
        with conn.cursor() as cur:
            cur.execute("UPDATE users SET name = %s WHERE id = %s", (new_name, user_id))
        conn.commit()
    finally:
        conn.close()
    # Invalidate the cache for this user
    cache_key = f"user:profile:{user_id}"
    redis_client.delete(cache_key)
    print(f"Cache key {cache_key} invalidated")

# Usage
update_user_name(42, "Alice Smith")
print(get_user_profile(42))  # Will miss cache and fetch fresh data

Expected output:

Cache key user:profile:42 invalidated
Cache MISS — fetching from DB
{'id': 42, 'name': 'Alice Smith', 'email': 'alice@example.com'}

Compare options / when to choose what

Pro tip: Start with cache-aside. It's the easiest to get right. Only move to write-through if you have a strong consistency requirement and can tolerate slower writes.

Troubleshooting & edge cases

Common mistakes

1. Forgetting to set a TTL — Cache entries live forever, wasting memory and guaranteeing stale data.
2. Using the same cache key for different queries — For example, using "user:42" for both profile and settings data. Always include a namespace.
3. Not handling serialization properly — Redis stores bytes, so you must json.dumps() before storing and json.loads() after retrieving. For complex Python objects, consider pickle, but beware of security risks.
4. Invalidating on every read — Don't delete a key unless the underlying data actually changed. Doing so defeats the purpose of caching.
5. Ignoring Redis connection failures — If Redis is down, your app should fall back to querying the DB directly, not crash.

Edge cases

• Cache stampede: When many concurrent requests all miss the cache and hit the DB simultaneously. Mitigate by using locks or probabilistic early expiration (e.g., Redis SETNX for locking, or setting a shorter TTL and refreshing).
• Large query results: Storing a 10 MB JSON blob in Redis will blow up memory. Keep cached values small (under 100 KB). Consider pagination or partial data.
• TTL tuning: Too short — high DB load. Too long — stale data. Monitor cache hit rate in production and adjust.

What you learned & what's next

You now understand how to cache database queries with Redis using the cache-aside pattern. You saw how to:

• Build a unique cache key from query parameters.
• Check Redis before hitting the database.
• Store serialized results with a TTL using setex.
• Invalidate the cache when data changes with delete.

You're ready to apply this pattern to any read-heavy part of your application — user profiles, product listings, configuration data, and more.

Next up: In the next lesson, you'll explore caching for Python functions with functools.cache and how Redis can serve as a distributed function cache — taking this concept global across multiple app instances.

Keep your Redis instance warm and your TTLs short!