Semantic Search Engine

Keyword matching gets you only so far. Search for "budget-friendly electric car" on a system that maps words literally, and you'll see every result for car and electric — but nothing about affordability or zero-emission vehicles. That's the pain point this lesson tackles head-on. By the end, you'll have built a semantic search engine that understands meaning, not just string matching, using Python and sentence embeddings.

The problem this lesson solves

Traditional search engines — think SQL LIKE queries or full-text indexes — treat text as a bag of characters. They fail when the query and the document use different words to express the same idea. This is called the vocabulary mismatch problem, and it's everywhere:

• A user searches for "how to fix a leaky faucet" but the document says "repair a dripping tap".
• A query for "affordable electric vehicles" misses a article about "low-cost EV models".
• A search for "heart attack symptoms" doesn't surface pages about "myocardial infarction warning signs".

For years, engineers worked around this with synonyms, stop-word removal, and hand-tuned scoring. That worked, barely, but it was brittle. A better approach has emerged: semantic search, which uses neural embeddings to capture meaning. Instead of matching literal strings, you match concepts.

Semantic search is not a luxury anymore — it's a requirement. Customers expect search boxes that "just get them." Internal tools use it to surface relevant documents. Recommendation systems built on it power entire product lines. If you're building an AI application, semantic search is the backbone of retrieval-augmented generation (RAG) — and if you're working with LLMs, you will soon need it.

Core concept / mental model

Think of embeddings as a translator that converts words and sentences into a universal language of numbers — specifically, a list of floating-point numbers called a vector.

Here's the mental model:

• Think of space. Imagine a giant 3D room where every sentence is a dot. Sentences about cars cluster near each other; sentences about cooking cluster somewhere else. The distance between dots reflects how related the ideas are.
• In practice, that space has 384 to 1536 dimensions (not just 3), but the idea holds. Semantically similar sentences have vectors that are close together (small distance) or point in the same direction (high cosine similarity).
• The engine's job: turn your search query into a vector, then find the document vectors most similar to it.

Cosine similarity is the most common way to measure that closeness. It computes the cosine of the angle between two vectors. A value of 1.0 means identical direction; 0.0 means orthogonal (unrelated); negative means opposite meaning.

Why cosine over raw distance? Because it ignores vector magnitude, so it's robust to different sentence lengths. A long document and a short query can still be compared fairly.

The core workflow of a semantic search engine looks like this:

Documents -> [Embedding model] -> Document vectors -> [Index] 
Query -> [Embedding model] -> Query vector -> [Similarity search] -> Ranked results

You encode documents once (offline), store the vectors, and at query time you embed the query and compare it against the stored vectors. That's the essence of building a semantic search engine.

How it works step by step

Let's break the process into clear stages:

1. Choose an embedding model. You need a model that turns text into vectors. Good options: sentence-transformers/all-MiniLM-L6-v2 (fast, small, 384 dims), BAAI/bge-small-en (strong performance), or OpenAI's text-embedding-3-small for hosted, high-performance vectors.

2. Prepare your corpus. Collect the documents you want to search. They could be product descriptions, FAQs, support tickets, or research papers. Each document should be a string of text.

3. Encode documents. Pass each document through the embedding model to produce a vector. Store these vectors in a list or a proper vector index.

4. Index the vectors. For small datasets (under ~100k rows), a simple in-memory list is fine. For scale, use a library like FAISS, Annoy, or a database with vector support (e.g., pgvector, ChromaDB).

5. Encode the query. At search time, embed the user's query using the same model. Critical: never mix models.

6. Compute similarity. For each document vector, calculate the cosine similarity with the query vector.

7. Rank and return. Sort by similarity score descending, and return the top-K results.

A critical nuance: always use the same model for documents and queries. If you embed queries with a different model, the vector spaces won't align, and your similarity scores will be meaningless.

Performance trick: for very large corpora, use an approximate nearest neighbor (ANN) index like FAISS. It trades a tiny bit of accuracy for massive speed gains.

Hands-on walkthrough

Let's build a working semantic search engine from scratch. We'll use sentence-transformers and NumPy.

First, install the dependencies:

pip install sentence-transformers numpy

Now, let's build the engine:

import numpy as np
from sentence_transformers import SentenceTransformer

# 1. Load a lightweight, fast embedding model
model = SentenceTransformer('all-MiniLM-L6-v2')

# 2. Corpus of documents — maybe a mini knowledge base
documents = [
    "Python is a high-level programming language known for its readability.",
    "TypeScript adds static typing to JavaScript, improving developer productivity.",
    "SQL is used to query and manage relational databases.",
    "Docker containerizes applications for consistent deployment.",
    "Kubernetes orchestrates containers at scale.",
]

# 3. Encode documents into vectors
doc_vectors = model.encode(documents, normalize_embeddings=True)

# 4. Define a query
query = "Which language is used for database queries?"
query_vector = model.encode([query], normalize_embeddings=True)[0]

# 5. Compute cosine similarity (dot product works because we normalized)
similarities = np.dot(doc_vectors, query_vector)

# 6. Rank and show top-3 results
ranked_indices = np.argsort(similarities)[::-1][:3]

print("Query:", query)
for idx in ranked_indices:
    print(f"Score: {similarities[idx]:.4f} | {documents[idx]}")

Expected output (scores will vary slightly):

Query: Which language is used for database queries?
Score: 0.7821 | SQL is used to query and manage relational databases.
Score: 0.1123 | Docker containerizes applications for consistent deployment.
Score: 0.0971 | Python is a high-level programming language known for its readability.

Notice how SQL ranks first, even though the word "SQL" appears in the document and the query uses "database queries" — the model captures the semantic relationship.

Now, let's handle larger datasets with FAISS for performance:

import numpy as np
import faiss
from sentence_transformers import SentenceTransformer

model = SentenceTransformer('all-MiniLM-L6-v2')
documents = [...]  # your corpus

# Normalize and convert to float32
vectors = model.encode(documents, normalize_embeddings=True)
vectors = np.float32(vectors)

# Build FAISS index
index = faiss.IndexFlatIP(vectors.shape[1])
index.add(vectors)

# Search
def search(query, k=3):
    q = np.float32(model.encode([query], normalize_embeddings=True))
    distances, indices = index.search(q, k)
    return [(documents[i], distances[0][j]) for j, i in enumerate(indices[0])]

results = search("tools to manage containers")
for doc, score in results:
    print(f"{score:.4f}: {doc}")

This scales to hundreds of thousands of documents with sub-millisecond latency.

Pro tip: always normalize_embeddings=True when using cosine similarity — it speeds up computation and avoids bugs with magnitude differences.

Compare options / when to choose what

Not all semantic search pipelines are created equal. Here's a comparison of common approaches:

When to choose what:

• For a tutorial or internal tool with a few hundred docs, start with NumPy.
• For a production service with >10k documents, go with FAISS.
• If you need to combine with metadata (e.g., filter by category), ChromaDB or pgvector give you an easy path.
• If you already run Elasticsearch, its vector feature might be the fastest path to integration.

Pro tip: Don't prematurely optimize. Get your brute-force version working first, measure latency, then switch to an ANN index only if you need it.

Troubleshooting & edge cases

Building a semantic search engine can come with surprising pitfalls. Here are the most common ones and how to fix them:

1. Similarity scores all near zero - Cause: You used different models for documents and query, or forgot to normalize. - Fix: Ensure model.encode(..., normalize_embeddings=True) for both, and use the same model object.

2. Search returns irrelevant results - Cause: The corpus is too domain-specific for a general model. - Fix: Fine-tune the embedding model on your own data, or choose a domain-specific model (e.g., BAAI/bge-small-en works well for English, microsoft/codebert-base for code).

3. MemoryError with large corpora - Cause: Storing all vectors in memory as Python lists/arrays. - Fix: Use a numpy array or FAISS index, and consider processing in batches.

4. Different results between runs - Cause: Non-deterministic operations or lack of seed. - Fix: Set a seed for reproducibility: np.random.seed(42) and torch.manual_seed(42) (if using PyTorch).

5. Empty query or very short query - Cause: The embedding vector for an empty string can be degenerate. - Fix: Preprocess queries — strip whitespace, handle empty strings gracefully (e.g., return all docs or a generic response).

6. Query contains out-of-vocabulary words - Cause: Model never saw the tokens. - Fix: Use a deep enough model with a large vocabulary; for extreme cases, add a fallback to keyword search.

What you learned & what's next

You've taken a huge step: you now know how to build a semantic search engine from scratch. Let's recap what you accomplished:

• You understood the vocabulary mismatch problem and why lexical search fails on meaning.
• You grasped the mental model of embeddings — mapping text to vectors in high-dimensional space.
• You implemented document encoding, query encoding, and cosine similarity to rank results.
• You compared different indexing strategies (NumPy, FAISS, ChromaDB, pgvector) and learned when to choose each.
• You troubleshooted common issues like bad similarity scores and memory overflows.

You've met all the learning objectives: you can explain the core idea and complete a hands-on exercise.

What's next? In the next lesson, you'll likely integrate semantic search into a retrieval-augmented generation (RAG) pipeline, adding an LLM to generate answers based on the retrieved passages. That's where semantic search truly shines — powering chatbots and knowledge assistants.

Keep practicing: try encoding a dataset of your own, like your notes or product docs, and query it with natural language. Measure how many relevant results pop up. You're now equipped to build systems that understand not just keywords — but intent.