Build a Recommendation Engine in Python with Collaborative Filtering

You’ve built the product; now you need people to actually use it. But every user is different. So you hand-craft personalized recommendations for each one—for, say, your first hundred users. Then your user base grows to a thousand. Then ten thousand. Now you’re trapped in a spreadsheet nightmare.

The solution? Build a recommendation engine. And not one powered by a black-box API—but one you understand completely, from math to code.

Here’s how to roll your own recommendation engine in Python using collaborative filtering, the simplest approach that still works shockingly well.

---

The Idea: “People Like You Also Liked…”

Collaborative filtering is the engine behind “Customers who bought this also bought…” on Amazon and “Because you watched…” on Netflix. The core insight is beautifully simple:

If user A and user B have similar tastes in the past, then user A will probably like something that user B liked and user A hasn’t tried yet.

No complex NLP. No neural networks. Just user behavior and math.

---

The Data You Need (Minimum)

Your engine needs just three pieces of information:

• user_id — who did it
• item_id — what they interacted with
• rating — how much they liked it (1–5 stars, play count, purchase, etc.)

That’s it. A CSV with three columns is enough to start.

---

Step 1: Build Your User-Item Matrix

First, turn your raw interactions into a matrix. Rows are users, columns are items, and cells are ratings.

import pandas as pd

df = pd.read_csv('ratings.csv')
matrix = df.pivot_table(index='user_id', columns='item_id', values='rating')

Most cells will be NaN — that’s the whole point. Your engine will predict those missing values.

---

Step 2: Find Similar Users (User-Based Filtering)

Now the math. To compare two users, you need a similarity measure. The classic choice is cosine similarity — it ignores differences in how generously people rate (a 4 from a generous rater might equal a 3 from a strict one).

from sklearn.metrics.pairwise import cosine_similarity

# Fill NaN with 0 for comparison, then compute similarity
similarity_matrix = cosine_similarity(matrix.fillna(0))
similarity_df = pd.DataFrame(similarity_matrix, index=matrix.index, columns=matrix.index)

Now for a given user, you find the top N most similar users who have actually rated the item in question.

---

Step 3: Predict Ratings — Weighted Average

For an item a user hasn’t seen, take the ratings from their most similar users, weighting each rating by how similar that user is.

def predict_rating(user_id, item_id, similarity_df, matrix, n_neighbors=10):
    # Get all users who rated this item
    other_ratings = matrix[item_id].dropna()

    # Get similarity scores for our target user
    sim_scores = similarity_df[user_id][other_ratings.index]

    # Take top N neighbors
    top_neighbors = sim_scores.sort_values(ascending=False)[:n_neighbors]

    # Weighted average
    numerator = sum(top_neighbors * other_ratings[top_neighbors.index])
    denominator = sum(top_neighbors)

    return numerator / denominator if denominator != 0 else 0

That’s your recommendation engine. Feed it a user, scan all unwatched items, rank by predicted rating, and serve the top 10.

---

One Problem: Cold Start

Your engine can’t recommend anything for a brand-new user with zero interactions. Solutions:

• Hybrid approach: Combine collaborative filtering with content-based filtering (e.g., recommend by item category)
• Popularity fallback: Serve the most popular items until the user generates enough data
• Explicit onboarding: Ask new users to rate a few items right away

---

Scaling Up (Because Your Original Matrix Will Be Sparse)

The method above is fine for a small dataset, but real-world matrices are 99%+ empty — and storing a dense similarity matrix for millions of users is a nightmare.

Enter matrix factorization — the technique behind the Netflix Prize winner. Libraries like scikit-learn and implicit handle this with:

from sklearn.decomposition import TruncatedSVD

svd = TruncatedSVD(n_components=20)  # 20 latent features
reduced_matrix = svd.fit_transform(matrix.fillna(0))

You can reconstruct predicted ratings from the reduced factors, dramatically saving memory and often improving accuracy.

---

What Real Teams Actually Use

For production, no one writes cosine similarity from scratch. The go-to tools are:

• Surprise (scikit-learn's recommender library) — great for prototyping
• Implicit — built for large-scale, sparse data (Amazon, Spotify scale)
• LightFM — hybrid collaborative + content features

But even with these libraries, you now understand what’s happening under the hood. And that understanding lets you tune, debug, and explain why a recommendation was made—which is more than most engineers can do.

---

The Bottom Line

Collaborative filtering is old tech. It predates modern deep learning by decades. But it still works. In many cases, a well-tuned nearest-neighbors approach beats a half-assed neural network.

Start with the three-column CSV, implement the weighted average, and watch your users discover things they actually like. Then worry about scaling. Because the hardest part isn’t the math—it’s knowing you built something that works from scratch.