Evaluate Recommenders with Precision@k

You've built a recommender that returns top-10 movie picks for every user. But when you demo it to stakeholders, the first question is: "How good is it?" Precision@k is the answer. It's the single most-used metric for recommender evaluation in production — the one that tells you, out of the top k recommendations you surface, how many are actually relevant. In this lesson, you'll learn how to compute precision@k, interpret it, and avoid the pitfalls that make naive implementations quietly lie to you.

The problem this lesson solves

Recommendation systems are easy to build and hard to trust. A model can return 10 items per user, but if only one of those items gets clicked, your system is essentially guessing. Without a clear evaluation metric, you're flying blind — you can't compare versions, tune hyperparameters, or convince anyone that your work matters.

The classic mistake is evaluating on offline accuracy — how well your model predicts ratings — and calling that success. But recommenders live in a messy world: users ignore 99% of what they're shown, and the items they do engage with are a tiny, biased sample. Precision@k cuts through that by focusing on the top of your recommendation list — the only part users actually see. It answers: of the top k items I surface, what fraction are relevant?

Without precision@k, you'll over-invest in models that look great in validation curves but fail in production. With it, you get a single number that aligns your team on what "good" means — and it's simple enough to explain to any stakeholder.

Core concept / mental model

Precision@k is a rank-aware precision metric — it measures the share of relevant items in the top k positions of your recommendation list.

Formula: Precision@k = (number of relevant items in top k) / k

Mental model: Think of precision@k as a mark on a dartboard. Your recommendation list is the dartboard; the first item you recommend is the bullseye, the second is the ring around it, and so on. Precision@k asks: if I only count the first k darts I throw, how many hit the target? It doesn't punish you for missing darts further out — because your user probably won't see them anyway.

Key definitions:

• Relevant item: an item the user actually engaged with — clicked, purchased, watched, or otherwise liked. This comes from your ground-truth data (e.g., a holdout set of real interactions).
• Top k: the first k items in your ranked recommendation list, usually k = 5 or 10.
• Rank-aware vs. rank-blind: precision@k is rank-aware because it only looks at the top of the list. Rank-blind metrics (like overall precision) look at your entire output — which is misleading when you recommend hundreds of items.

Why it matters: If your recommender returns 10 items and only 3 are relevant, precision@10 = 0.3. If you only show the user 5 of those items and 3 are relevant, precision@5 = 0.6 — a much better story. Precision@k rewards you for putting the best items first, which is exactly what a recommender should do.

How it works step by step

Here's the step-by-step recipe for computing precision@k:

1. Get ground truth per user. For each user, have a set of relevant items from your test data (e.g., interactions that happened after your model was trained).
2. Generate recommendations per user. Run your model to produce a ranked list of items for each user.
3. Take the top k items of each recommendation list.
4. Count the relevant ones in that slice. Intersect the top k with the ground-truth set.
5. Divide by k to get the per-user precision@k.
6. Aggregate across users — usually by taking the mean — to get your overall precision@k.

Why you need to be careful about ground truth

Your ground-truth set is only as good as your data. If you mark every item a user could have liked as relevant, your precision@k will be artificially high. Conversely, if your ground truth is sparse (e.g., you only record purchases, not views), you'll underestimate relevance. Always understand how your relevance labels are defined before trusting the metric.

The role of k

Choosing k is a product decision. If you show 5 items on a mobile screen, use k=5. If your UI displays 20 results, use k=20. Smaller k emphasizes the very top of your list — more about ranking quality — while larger k measures overall coverage. In practice, most teams report both precision@5 and precision@10.

Hands-on walkthrough

Let's compute precision@k in Python with a concrete example. We'll start with a simple toy dataset, then scale up to a more realistic pattern.

Example 1: Single-user precision@k

from typing import List, Set

def precision_at_k(
    recommended: List[str],
    relevant: Set[str],
    k: int
) -> float:
    """Compute precision@k for one user."""
    if k <= 0:
        raise ValueError("k must be positive")
    top_k = recommended[:k]
    relevant_in_top_k = sum(1 for item in top_k if item in relevant)
    return relevant_in_top_k / k

# Example: user liked horror movies and one drama
relevant = {"The Shining", "Hereditary", "Midsommar", "Get Out"}
recommended = ["The Shining", "La La Land", "Hereditary", "Toy Story", "Parasite"]

print(f"Precision@5: {precision_at_k(recommended, relevant, 5):.2f}")
print(f"Precision@3: {precision_at_k(recommended, relevant, 3):.2f}")
# Output:
# Precision@5: 0.40
# Precision@3: 0.67

Notice how precision@3 is higher — the two relevant items happen to sit in the top 3. Rank matters.

Example 2: Mean precision@k across users

import numpy as np
from typing import List, Set, Dict

def mean_precision_at_k(
    recommendations: Dict[str, List[str]],
    relevant_items: Dict[str, Set[str]],
    k: int
) -> float:
    """Compute mean precision@k across all users."""
    precisions = []
    for user, rec_list in recommendations.items():
        if user not in relevant_items:
            continue
        precisions.append(
            precision_at_k(rec_list, relevant_items[user], k)
        )
    return float(np.mean(precisions)) if precisions else 0.0

# Toy dataset: 3 users
recommendations = {
    "alice": ["item_a", "item_b", "item_c", "item_d", "item_e"],
    "bob":   ["item_x", "item_a", "item_b", "item_c", "item_f"],
    "carol": ["item_b", "item_d", "item_e", "item_a", "item_g"],
}

relevant_items = {
    "alice": {"item_a", "item_b"},
    "bob":   {"item_a", "item_f", "item_x"},
    "carol": {"item_d", "item_e"},
}

print(f"Mean precision@5: {mean_precision_at_k(recommendations, relevant_items, 5):.2f}")
# Expected output: 0.47 (alice: 0.4, bob: 0.6, carol: 0.4 -> /3)

This aggregated number is what you'd report to stakeholders. It gives you a single, comparable number for model A vs. model B.

Example 3: Bonus — precision@k with a placeholder for missing items

When your model can't produce a full list of k items (e.g., a cold-start user), you should decide how to handle it. The common approach is to pad with empty strings and count them as non-relevant.

def precision_at_k_filled(
    recommended: List[str],
    relevant: Set[str],
    k: int
) -> float:
    """Compute precision@k, padding short lists with 'PAD'."""
    padded = recommended + ["PAD"] * (k - len(recommended))
    top_k = padded[:k]
    relevant_in_top_k = sum(1 for item in top_k if item in relevant)
    return relevant_in_top_k / k

# User with only 2 recommendations
short_recs = ["item_a", "item_b"]
relevant = {"item_a"}
print(f"Precision@5 (unfilled): {precision_at_k(short_recs, relevant, 5):.2f}")
print(f"Precision@5 (filled):   {precision_at_k_filled(short_recs, relevant, 5):.2f}")
# Output:
# Precision@5 (unfilled): 0.20
# Precision@5 (filled):   0.20
# Both are same here *because* we're dividing by k, not by list length.

Pro tip: Always divide by k, not by the length of your recommendation list. This makes precision@k comparable across users who might receive different numbers of items — otherwise you'd inflate scores for users with shorter lists.

Compare options / when to choose what

Precision@k isn't the only metric — and it's not always the best. Here's how it stacks up against common alternatives:

When to choose precision@k: - You're showing a fixed-size list to users (top-5, top-10). - You want a metric that non-technical stakeholders can grasp instantly. - You want to debug ranking quality quickly (e.g., compare precision@1 to precision@10).

When to skip it: - If your UI shows all recommendations (no k-limit), use overall precision or recall. - If you need to evaluate ranking order, not just presence, NDCG is better — precision@k treats every position in the top k equally.

Blockquote: "Precision@k is a necessary but not sufficient metric for recommender health. Pair it with coverage, diversity, and a business metric (e.g., CTR) before shipping." — common industry advice

Troubleshooting & edge cases

Precision@k is > 1.0

Cause: You divided by the number of relevant items instead of k.

Fix: Always divide by k. If you have 5 relevant items in top 10, precision@10 = 0.5, never 5.0.

Precision@k is identical for two very different models

Cause: Both models put the same relevant items in the top k, but in different orders. Precision@k is order-insensitive within the top k.

Fix: If ordering matters, switch to NDCG@k or mean reciprocal rank (MRR). Precision@k only cares about whether an item is in the top k, not where.

Users with no relevant items in ground truth

Cause: Some users have zero relevant interactions in your test set — typical for new users.

Fix: Decide upfront: either exclude them (reducing your user base) or include them with precision@k = 0 (punishing the model unfairly). In practice, including them with 0 is the standard approach, but always report the proportion of users you had to exclude.

Short recommendation lists (fewer than k items)

Cause: Cold-start users or sparse catalogs.

Fix: Pad with placeholders as in Example 3, or compute precision@k only up to the number of available items and divide by that number (but be transparent). The key is consistency across all users.

Bias in ground-truth collection

Cause: You marked items as relevant if the user saw them, but not necessarily liked them. This inflates precision@k because your model is rewarded for recommending popular items.

Fix: Use implicit feedback (clicks, purchases) rather than exposure. If you only have exposure data, you're measuring "accuracy of what was shown," not "quality of what was recommended."

What you learned & what's next

You now understand how to evaluate recommenders with precision@k:

• The definition — relevant items in top k divided by k.
• The mental model — dartboard: top of the list matters most.
• The implementation — a few lines of Python, with sensible handling of edge cases.
• The comparison — where precision@k beats recall@k and where NDCG is a better fit.
• The pitfalls — from dividing by the wrong denominator to ignoring order-sensitivity.

You can now explain precision@k to anyone, compute it confidently, and choose the right metric for your evaluation harness. In the next lesson, you'll extend this to recall@k and F1@k — combining precision with coverage to get a fuller picture of your recommender's health. With that, you'll be ready to build a complete evaluation suite for your own recommendation system.