Cache Predictions with Redis

You've built a brilliant model. It returns dazzling predictions with high accuracy. But every time a user hits your endpoint, the model runs the same expensive computation or API call over and over — burning GPU cycles, inflating your cloud bill, and adding latency that drives users away. The pain is real: you need speed and efficiency without sacrificing quality. This lesson shows you how to cache predictions with Redis, a battle-tested in-memory data store, to slash response times and costs while keeping your AI application snappy and scalable.

The problem this lesson solves

When every prediction repeats the same work, your application suffers in three ways:

• Latency: Model inference is often the slowest part of your request pipeline. Returning a cached result in microseconds beats recalculating in milliseconds or seconds.
• Cost: Every uncached prediction consumes compute (GPU/CPU) or calls a paid API. Duplicate work means wasted money.
• Load: High traffic can overwhelm your model serving infrastructure, causing errors and timeouts.

Caching predictions with Redis directly addresses all three by storing the result of a prediction keyed by its input. The next time the same request comes in, you serve the stored answer instead of recomputing it. This is a cornerstone of applied AI engineering — optimizing systems for production reality, not just model quality.

Core concept / mental model

Think of Redis as an ultra-fast lookup table sitting in front of your model. It's like a librarian who remembers the answer to a frequently asked question instead of fetching the book every time.

• Key: A unique string representing the prediction request (e.g., a hash of the input features or prompt).
• Value: The serialized prediction result (e.g., a JSON string).
• TTL (Time to Live): An expiry time so cached predictions don't go stale.

Why Redis? It's an in-memory data store with sub-millisecond read/write speeds, supports data structures like strings and hashes, and has built-in expiration. It's designed for exactly this job.

How it works step by step

1. Generate a cache key from the input. For deterministic models, a hash of the input works well. For LLMs, include the full prompt and parameters (temperature, max tokens, etc.).
2. Check Redis for the key. If found, return the cached value immediately — this is a cache hit.
3. If not found (a cache miss), run the model to get the prediction.
4. Store the prediction in Redis with a TTL, so future identical requests can be served faster.
5. Return the prediction to the caller.

The cause-effect chain is simple: the first request pays the full inference cost, but subsequent identical requests get near-instant responses.

Hands-on walkthrough

Let's implement a prediction caching layer in Python. We'll use redis-py and a dummy model function.

Step 1: Install and connect

pip install redis

import redis
import hashlib
import json
import time

# Connect to Redis (default localhost:6379)
r = redis.Redis(host='localhost', port=6379, db=0)

Step 2: Create a caching decorator

def cache_predictions(ttl_seconds=3600):
    """Decorator that caches predictions in Redis."""
    def decorator(func):
        def wrapper(*args, **kwargs):
            # Create a unique key from the input arguments
            key_material = json.dumps({'args': args, 'kwargs': kwargs}, sort_keys=True, default=str)
            cache_key = f"pred:{func.__name__}:{hashlib.sha256(key_material.encode()).hexdigest()}"

            # Check cache
            cached = r.get(cache_key)
            if cached is not None:
                print("Cache HIT")
                return json.loads(cached)

            print("Cache MISS — computing...")
            result = func(*args, **kwargs)
            # Store result with TTL
            r.setex(cache_key, ttl_seconds, json.dumps(result))
            return result
        return wrapper
    return decorator

Step 3: Use it with a model

@cache_predictions(ttl_seconds=60)
def predict_house_price(sqft, bedrooms):
    # Simulate expensive model inference
    time.sleep(2)  # pretend it takes 2 seconds
    return {'price': 1000 * sqft + 5000 * bedrooms, 'timestamp': time.time()}

# First call — miss, takes ~2 seconds
print(predict_house_price(1500, 3))

# Second call — hit, instant
print(predict_house_price(1500, 3))

Expected output:

Cache MISS — computing...
{'price': 1650000, 'timestamp': 1700000000.0}
Cache HIT
{'price': 1650000, 'timestamp': 1700000000.0}

Notice the cached value still has the original timestamp — that's important: you're serving the exact same prediction, not a fresh one.

Step 4: Handle varying inputs

Your cache key must capture everything that affects the prediction:

def predict_sentiment(text, model_version='v1'):
    key_material = json.dumps({'text': text, 'version': model_version}, sort_keys=True)
    cache_key = f"sentiment:{hashlib.sha256(key_material.encode()).hexdigest()}"
    # ... rest of logic

If you miss a parameter (like model_version), you'll serve stale or wrong predictions.

Compare options / when to choose what

Pro tip: Start with an in-memory cache for prototyping, but move to Redis as soon as you have more than one server or need shared state.

Troubleshooting & edge cases

• Redis connection refused: Ensure Redis server is running (redis-server). Check host/port.
• Key collision: Use a secure hash (SHA-256) and include all relevant parameters. Avoid simple string concatenation.
• Cached predictions go stale: Set appropriate TTLs. For models that update, include a version in the key or invalidate the cache on model deployment.
• Serialization errors: Use json.dumps with default=str for non-serializable objects. For complex objects, consider pickle, but be careful with security.
• Memory blowup: Set a maxmemory policy in Redis (e.g., allkeys-lru) and use TTLs.
• Race conditions: Two identical requests can both miss the cache and compute twice. Use Redis locks (e.g., set nx) to prevent this if it's critical.

What you learned & what's next

You've mastered the core idea of caching predictions with Redis: store model outputs keyed by input to avoid recomputation. You've applied it in a hands-on decorator, learned to compare caching strategies, and know how to troubleshoot common issues. You can now explain how caching reduces latency and cost, and you've completed a practical exercise — you're ready to apply this pattern to your own AI services.

Next in the track: handling dynamic prompts and context — where you'll learn to design cache keys for LLM chat systems that respect conversation history and personalization.