Build a RAG Chat Assistant

We've all seen chat assistants that sound confident but hallucinate answers, or that can't answer questions about your own documents. Building a chat assistant with a RAG pipeline is the industry-standard way to fix that: you ground every answer in your own data, so responses are accurate, up-to-date, and traceable. In this lesson, you'll learn how to build a chat assistant with a RAG pipeline from scratch, using Python 3.10+ and practical, hands-on steps you can apply immediately.

The problem this lesson solves

Without retrieval, your chat assistant is just a general-purpose language model. It has no access to your private documents, your latest product specs, or your internal knowledge base. The result: it makes things up, it gives stale answers, and it can't cite its sources.

Consider a support bot for a SaaS product. The model may know the internet's version of your API, but it won't know that you renamed a parameter last Tuesday. Users get frustrated, trust erodes, and your team ends up rewriting answers by hand.

RAG solves this by splitting the problem: first retrieve relevant chunks from your data, then generate an answer grounded in those chunks. This is the core of building a chat assistant with a RAG pipeline — and it's why every enterprise AI team uses it.

Here's what you'll achieve by the end of this lesson:

• Explain the core idea behind building a chat assistant with RAG pipeline
• Complete a practical exercise that builds a working RAG assistant
• Connect this to the next step in your Applied AI engineering track

Core concept / mental model

Think of RAG like a reference desk at a library. You don't ask the librarian (the LLM) to recite every book from memory — that would be unreliable. Instead, you tell the librarian your question, they walk to the shelves (your vector database), pull the most relevant books (retrieved chunks), and then hand you a synthesized summary with page numbers (citations).

In technical terms, RAG is Retrieval-Augmented Generation. The pipeline has four parts:

1. Ingestion: you split your documents into chunks and embed them into vectors.
2. Storage: you store those vectors in a vector database (e.g., Chroma, FAISS, or Pinecone).
3. Retrieval: given a user query, you embed it and find the most similar chunks via similarity search.
4. Generation: you pass the retrieved chunks as context to an LLM, which answers the query based solely on that context.

Definitions you'll need

• Chunk: a small slice of text (200–1000 characters) that's self-contained enough to answer a question.
• Embedding: a numeric vector that captures the meaning of a text; similar texts have similar vectors.
• Top-k: the number of chunks you retrieve for each query.
• Context window: how much text the LLM can see in one prompt.

Why RAG beats fine-tuning for this use case

Fine-tuning changes the model's weights — expensive, slow, and it still can't guarantee accuracy on private data. RAG keeps the model frozen and swaps out the context. This means:

• Fresh data: update your vector store without retraining.
• Traceability: you can show the exact source chunks.
• Cost: no GPU training needed, just API calls.

How it works step by step

Building a chat assistant with a RAG pipeline follows a repeatable sequence. Let's walk through each stage so you understand the why before you write the code.

Step 1: Prepare your documents

Gather all the text you want the assistant to know about: manuals, FAQs, Slack exports, academic papers. The quality of your chunks decides the quality of your answers, so start by cleaning the data — remove boilerplate, fix encoding, and split long files into coherent sections.

Step 2: Chunk the text

You can't embed an entire 200-page PDF in one vector — it would be impossible to retrieve a specific answer. Instead, split your text into chunks with a smart strategy:

• Fixed-size: split every N characters with overlap. Simple, but may cut sentences in half.
• Recursive character: split on paragraph or sentence boundaries. Better for prose.
• Semantic: use the model to identify natural breaks. Best, but more API calls.

A good chunk size is 300–800 characters with an overlap of 10–20%. This keeps context coherent while still allowing precise retrieval.

Step 3: Embed and store

Embed each chunk using a model like text-embedding-ada-002 (OpenAI) or all-MiniLM-L6-v2 (open-source). Store the vectors in a vector database. Here's the flow:

from sentence_transformers import SentenceTransformer
import chromadb

# Load a free, local embedding model
model = SentenceTransformer('all-MiniLM-L6-v2')

# Create a Chroma client and collection
client = chromadb.PersistentClient(path="./my_db")
collection = client.get_or_create_collection(name="docs")

chunks = [
    "RAG stands for Retrieval-Augmented Generation. It combines retrieval with LLM generation.",
    "Chunking is the process of splitting documents into smaller pieces for embedding."
]
embeddings = model.encode(chunks).tolist()

# Add to the collection
collection.add(
    ids=["0", "1"],
    documents=chunks,
    embeddings=embeddings
)

Pro tip: You don't need to store embeddings manually if you use Chroma's built-in embedding function. But showing the explicit encoding helps you understand what's happening under the hood.

Step 4: Retrieve relevant chunks

When a user asks a question, you embed the query and search for the closest chunks. Most vector databases do this with cosine similarity or Euclidean distance. Choose a top_k that fits your context window — usually 3–5 chunks.

Step 5: Ground the LLM answer

You construct a prompt that includes the retrieved chunks and the user's question, with instructions to answer only from the context. This is where the magic happens — the LLM can't hallucinate because you've given it the exact material.

Hands-on walkthrough

Now let's put it together. We'll build a chat assistant that answers questions about a fictional product manual. You'll need Python 3.10+, the chromadb, sentence-transformers, and openai packages (or a local LLM like Llama 3.2 via Ollama).

Setup

pip install chromadb sentence-transformers openai

If you want to avoid API costs, install Ollama and pull a small model:

ollama pull llama3.2:3b

Full RAG pipeline

The following script does everything: ingests a few document chunks, builds the vector store, retrieves an answer, and prints the response with sources.

import chromadb
from sentence_transformers import SentenceTransformer
import os

# Use OpenAI API (or set to 'ollama' for local)
USE_OLLAMA = os.getenv("USE_OLLAMA", "false").lower() == "true"

if USE_OLLAMA:
    from langchain_community.chat_models import ChatOllama
    llm = ChatOllama(model="llama3.2:3b", temperature=0.0)
else:
    from openai import OpenAI
    client = OpenAI()

# 1. Chunk data
with open("product_manual.txt", "r") as f:
    text = f.read()
chunks = [text[i:i+500] for i in range(0, len(text), 250)]  # overlap

# 2. Embed + store
model = SentenceTransformer('all-MiniLM-L6-v2')
chroma_client = chromadb.PersistentClient(path="./manual_db")
collection = chroma_client.get_or_create_collection(name="manual")
collection.add(
    ids=[str(i) for i in range(len(chunks))],
    documents=chunks,
    embeddings=model.encode(chunks).tolist()
)

# 3. Retrieve
def retrieve(query, top_k=3):
    q_emb = model.encode(query).tolist()
    res = collection.query(query_embeddings=[q_emb], n_results=top_k)
    return res["documents"][0]

# 4. Generate
def ask(question):
    docs = retrieve(question)
    context = "\n\n".join(docs)
    prompt = f"""Answer the question using ONLY the context below. If the context doesn't suffice, say "I don't know".

Context:
{context}

Question: {question}
"""
    if USE_OLLAMA:
        return llm.invoke(prompt).content
    else:
        resp = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": prompt}]
        )
        return resp.choices[0].message.content

print(ask("What is the return policy?"))

Expected output (example):

Our return policy allows returns within 30 days of purchase, provided the product is unused and in original packaging.

A minimal example with LangChain (optional)

If you prefer higher-level abstractions, LangChain gives you a RetrievalQA chain:

from langchain.chains import RetrievalQA
from langchain_community.vectorstores import Chroma
from langchain_community.embeddings import HuggingFaceEmbeddings
from langchain_openai import ChatOpenAI

embedding = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")
vectorstore = Chroma(persist_directory="./manual_db", embedding_function=embedding)
qa = RetrievalQA.from_chain_type(
    llm=ChatOpenAI(model="gpt-4o-mini"),
    chain_type="stuff",
    retriever=vectorstore.as_retriever()
)

print(qa.run("How do I reset the device?"))

Pro tip: Keep the raw chunk text in your database and return source metadata along with the answer — it makes debugging and citations trivial.

Compare options / when to choose what

You have several choices when building a chat assistant with a RAG pipeline. The right one depends on your scale, budget, and pivot-tolerance.

Vector databases

Choose Chroma when you're learning or building a demo. Choose Pinecone or Weaviate when you need uptime, scaling, and built-in filtering.

Embedding models

• all-MiniLM-L6-v2 (local, free, 384 dims): good for English, limited multilingual.
• text-embedding-3-small (OpenAI, $0.02/1M tokens): fast, surprisingly good, supports dimensions tuning.
• BGE-M3 (local, 1024 dims): excellent multilingual, heavier.

LLM providers

• OpenAI GPT-4o-mini: cheap, reliable, strong instruction-following.
• Anthropic Claude: excels at long context and safety.
• Local models (Ollama): privacy, zero per-request cost, but need decent hardware.

Chunking strategies

When to choose what: For a production assistant, use recursive chunking with 10% overlap, a managed vector DB if you have >100k docs, and a cloud LLM for quality. For a local offline tool, use fixed-size + Chroma + Ollama.

Troubleshooting & edge cases

Even with a perfect pipeline, things go wrong. Here are the most common issues and how to fix them.

1. "I don't know" when the answer is in the docs

Symptom: The assistant says it doesn't know, but the chunk exists.

Cause: The chunk isn't being retrieved — either the query embedding is off, or top_k is too low.

Fix:

# Increase retrieval count
res = collection.query(query_embeddings=[q_emb], n_results=5)

Also, try a different embedding model. all-MiniLM may miss nuanced synonyms. Switch to text-embedding-3-small if you use OpenAI.

2. Hallucination even with RAG

Symptom: The answer includes facts not in the context.

Cause: The prompt isn't strict enough, or the retrieved chunks are irrelevant.

Fix: Re-read your prompt. Use explicit instructions:

prompt = "You are a strict assistant. If the context does not answer the question, say \"I don't know.\" Do NOT use outside knowledge."

Also, filter retrieved chunks by similarity threshold:

res = collection.query(query_embeddings=[q_emb], n_results=5, include=["distances"])
# Drop chunks with distance > 0.8 (cosine distance)

3. Query is too long or too short

Issue: Very long queries dilute the embedding signal; too short queries retrieve too many unrelated chunks.

Fix: Rephrase the query before embedding — either extract keywords or summarize the question with a quick LLM call.

4. Token limit exceeded in generation

Symptom: OpenAIError: This model's maximum context length is 8192 tokens.

Fix: Reduce top_k or chunk size. Calculate total tokens: (chunk_size * top_k) + question + prompt should be under the model limit.

top_k = 3
chunk_size = 400  # characters
# Rough estimate: 4000 tokens per 3000 chars — tune down

5. Vector database is empty or wrong results

Symptom: Retrieval returns empty or stale chunks.

Fix: Check that you're using the same collection name and path. Persistent Chroma can hold old data — delete and recreate when testing.

client.delete_collection("manual")
client.create_collection("manual")  # fresh start

6. Embedding dimension mismatch

Error: all embeddings must have the same dimension.

Fix: Stick to one embedding model for both ingestion and query. Never mix all-MiniLM with OpenAI embeddings.

What you learned & what's next

You've now built a chat assistant with a RAG pipeline from scratch. Let's recap what you mastered:

• The core concept: RAG = retrieval + generation, and why it grounds AI answers in your own data.
• The step-by-step workflow: chunking, embedding, storage, retrieval, and grounded generation.
• Hands-on implementation: you wrote a working pipeline using Chroma, an embedding model, and an LLM.
• Comparison skills: you know when to pick Chroma vs. Pinecone, local vs. cloud models, and chunking strategies.
• Troubleshooting: you can diagnose retrieval misses, hallucination, token limits, and dimension errors.

Your next step

Your chat assistant works, but can it handle multi-turn conversations? In the next lesson, you'll add conversation memory and advanced prompt engineering so the assistant understands follow-up questions like "What about the Pro version?" from earlier context. You'll also learn how to evaluate your RAG pipeline's retrieval quality — the difference between a demo and a production system.

Keep practicing. Every real-world AI engineer starts exactly where you are now.