Build a RAG system with ChromaDB

You've built a great RAG prototype in a notebook, but now you need to make it production-ready. The pain is real: a naive RAG pipeline can choke on thousands of documents, return irrelevant chunks, and hallucinate answers that undermine user trust. This lesson solves that by showing you how to build a RAG system with ChromaDB — a vector database that powers fast, scalable retrieval so your LLM answers with grounded, relevant context every time.

The problem this lesson solves

Retrieval-Augmented Generation (RAG) is the backbone of modern AI applications — from customer support bots to internal knowledge assistants. Yet many developers struggle with the retrieval half. A simple in-memory list of documents can't scale: every query scans everything, latency spikes, and your LLM receives stale or irrelevant context. The result? Poor answer quality and a system that doesn't generalize.

This lesson walks you through replacing that brittle approach with ChromaDB, a purpose-built vector store. By the end, you'll retrieve the right chunks in milliseconds and feed them to an LLM to generate grounded, accurate answers. You'll be able to build a RAG system with ChromaDB that handles hundreds of documents without breaking a sweat.

Core concept / mental model

Think of a RAG system as a research assistant with a photographic memory. The assistant reads your documents, files away every key point as a vector — a list of numbers that captures the meaning of the text. When you ask a question, the assistant searches its memory for the most similar vectors and retrieves the relevant passages. Finally, it drafts an answer using that context, not just its general knowledge.

So your RAG stack has four components: - Ingestion: Convert documents into text chunks. - Embedding: Turn text into vectors using a model like all-MiniLM-L6-v2. - Storage & retrieval: Store vectors in ChromaDB, then query with cosine similarity. - Generation: Feed the retrieved context to an LLM (e.g., OpenAI, Ollama) and get an answer.

ChromaDB shines because it's embedded, persistent, and simple — no separate database server needed. It handles batching, metadata filtering, and similarity searches right in your Python process.

How it works step by step

Let’s trace the data flow, from raw document to final answer:

1. Collect documents — Load PDFs, text files, or HTML pages into your app.
2. Chunk — Split long documents into overlapping segments (e.g., 200 tokens with 50 overlap). Overlap preserves context at boundaries.
3. Embed — Pass each chunk through an embedding model to get a numeric vector.
4. Store — Save vectors + original text + metadata into a ChromaDB collection.
5. Query — Embed the user’s question, then ask ChromaDB for the top-k most similar vectors (cosine similarity).
6. Generate — Send the retrieved passages as context in your LLM prompt, along with the question.
7. Return — The LLM’s answer is grounded in your documents.

Why cosine similarity instead of exact match? Semantic search matches meaning, not keywords. A typo or a synonym won’t break retrieval, and you can even retrieve text in a different language if your embedding model supports it.

Hands-on walkthrough

Let’s build it. First, install the required packages:

pip install chromadb sentence-transformers openai

Step 1: Ingest and index your documents

Create a Python script that reads a few text files, chunks them, and stores them in ChromaDB:

import chromadb
from sentence_transformers import SentenceTransformer
from pathlib import Path

# Initialize ChromaDB (persistent storage)
client = chromadb.Client()
collection = client.create_collection(name="company_docs")

# Embedding model
transformer = SentenceTransformer("all-MiniLM-L6-v2")

def ingest_file(filepath: str, chunk_size: int = 200, overlap: int = 50):
    text = Path(filepath).read_text()
    words = text.split()
    chunks = []
    for i in range(0, len(words), chunk_size - overlap):
        chunks.append(" ".join(words[i:i + chunk_size]))

    # Embed all chunks
    embeddings = transformer.encode(chunks).tolist()

    # Add to ChromaDB with metadata
    ids = [f"{filepath}_{i}" for i in range(len(chunks))]
    metadatas = [{"source": filepath} for _ in chunks]
    collection.add(ids=ids, documents=chunks, embeddings=embeddings, metadatas=metadatas)

ingest_file("hr_policy.txt")
ingest_file("onboarding_guide.txt")

print(f"Indexed {collection.count()} chunks")

Expected output:

Indexed 42 chunks

Step 2: Retrieve relevant context

Now query the collection to get the top 3 most similar chunks for a user question:

def retrieve(question: str, top_k: int = 3):
    q_embedding = transformer.encode(question).tolist()
    results = collection.query(query_embeddings=[q_embedding], n_results=top_k)
    return [doc for doc in results["documents"][0]]

question = "What is the company's remote work policy?"
chunks = retrieve(question)
for i, chunk in enumerate(chunks, 1):
    print(f"Chunk {i}: {chunk}")

Step 3: Generate an answer with an LLM

Feed those chunks to an LLM (here using OpenAI's API, but you can swap in Ollama or others):

from openai import OpenAI

client_llm = OpenAI()

def ask(question: str, context_chunks: list[str]) -> str:
    context = "\n\n---\n\n".join(context_chunks)
    response = client_llm.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "Answer using only the provided context. If uncertain, say 'I don't know'."},
            {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"}
        ]
    )
    return response.choices[0].message.content

print(ask(question, chunks))

Expected output:

Employees can work remotely up to three days per week, subject to manager approval, and must maintain a reliable internet connection.

Pro tip: In production, cache the embedding of frequent queries to avoid recomputing. Also, set n_results based on your LLM’s context window — all chunks must fit in the prompt.

Compare options / when to choose what

ChromaDB is one of many vector databases. Here’s how it stacks up against other popular choices:

If you’re building a quick proof-of-concept or an internal tool with under a million chunks, ChromaDB is perfect. For massive scale or managed infrastructure, consider Pinecone or Weaviate. For pure speed in research pipelines, FAISS is a solid choice.

Variations: Instead of the SentenceTransformer model shown here, you can use APIs like OpenAI’s text-embedding-3-small or Cohere’s embed models. The pipeline stays the same — you just plug in a different embedding function. Also, consider chunking strategies: fixed-size tokens vs. paragraph-based splitting. Paragraph-preserving chunks often yield better context.

Troubleshooting & edge cases

Empty results from ChromaDB

If your query returns nothing, it’s often because the embedding model is inconsistent. Always use the same model for indexing and querying. Also, check that the collection name matches; a typo creates a new empty collection.

Memory spikes during embedding

Embedding thousands of chunks at once can exhaust RAM. Solution: process in batches (e.g., 64 chunks per call) and add each batch to ChromaDB. This also speeds up ingestion since the DB doesn’t re-index everything.

Duplicate chunks

If you re-run the ingestion script, ChromaDB will throw a UniqueConstraintError because IDs already exist. Fix: either use a unique ID suffix based on timestamp, or delete the collection before re-adding.

client.delete_collection("company_docs")
collection = client.create_collection("company_docs")

Outdated or conflicting answers

Sometimes the retrieved chunks contradict each other. Mitigations: add a metadata field for document date and filter by it during query, or summarize multiple conflicting chunks before final generation.

Edge case: If your documents contain a lot of numbers or code, smaller chunk sizes (100–150 tokens) often help preserve logical structure.

What you learned & what's next

You now know how to build a RAG system with ChromaDB end-to-end. Specifically, you learned to: - Explain the core idea behind RAG and the role of ChromaDB as a vector store. - Complete a practical exercise — ingest, retrieve, and generate using ChromaDB and an LLM. - Choose the right vector DB based on your scale and ops needs. - Handle common pitfalls like empty results, memory spikes, and duplicate IDs.

Next in the track, you’ll dive into evaluation harnesses — how to measure answer accuracy and retrieval quality systematically. That’s where you’ll turn this working RAG system into a trustworthy product. Keep building!