Build a Q&A System with BERT

You have a pile of documents, a burning question, and no time to read every page. Manually extracting answers from long reports, support tickets, or research papers is slow, error-prone, and doesn't scale. This lesson shows you how to build a question answering system with BERT — a focused, extractive QA pipeline that pinpoints the exact sentence in your text that answers a user's question. By the end, you'll have a working Python system that turns raw documents into instant answers, and you'll understand the core ideas behind the transformer revolution in NLP.

The problem this lesson solves

Traditional search returns documents — you still have to read them. Keyword matching (think regex or simple in checks) fails at synonyms and paraphrase: asking "What is the capital of France?" won't match a paragraph that says "Paris serves as the seat of government." This is the information overload problem that every analyst, support team, and researcher faces daily.

You need a system that understands the meaning of your question and the context of the document, and then extracts the answer — not a list of links. A BERT-based question answering system does exactly this: it takes a question and a passage, and returns the start and end positions of the answer span within the passage. It's a supervised extractive QA approach, trained on pairs of questions and passages with labeled answers.

The pain is real: support teams drowning in ticket backlogs, legal teams scanning thousands of contract pages, researchers reviewing hundreds of papers. Automating the answer-finding step with BERT slashes time from hours to milliseconds, and it's accessible even on modest hardware using a pre-trained model.

Core concept / mental model

Think of BERT as a high-speed reader with a highlighter. When you give it a question and a passage (the context), it doesn't 'generate' text from scratch — it reads the passage, understands the question's intent, and highlights the exact substring that answers it. This is extractive question answering, as opposed to generative approaches (like GPT) that write new sentences.

Why BERT? BERT (Bidirectional Encoder Representations from Transformers) processes text both left-to-right and right-to-left simultaneously, capturing deep context. Each word (token) gets a rich vector representation that depends on all the other words around it. For QA, two special output layers are added: one predicts the probability that a token is the start of the answer, the other predicts the probability that it's the end. The system then selects the span with the highest combined probability.

How it works in three layers: 1. Tokenizer — converts raw text into token IDs, adding special [CLS] and [SEP] tokens to mark the question and passage boundaries. The input format is [CLS] question [SEP] passage [SEP]. 2. Encoder — passes token IDs through the transformer layers, producing a context-aware embedding for every token. 3. Classifier head — two linear layers that map each token embedding to start/end scores. The span with the highest total score becomes the answer.

This architecture is task-agnostic — you can fine-tune the same pre-trained BERT on any extractive QA dataset (like SQuAD) and it learns the pattern of highlighting relevant spans.

How it works step by step

Here's the cause-and-effect chain of an extractive QA pipeline:

1. Input preparation — You provide a question (string) and a context (string). The tokenizer combines them into a single sequence with special separators. If the context is long, it's split into overlapping chunks (sliding window) to stay within the model's maximum token limit (typically 512).
2. Tokenization — Words are broken into subword tokens (e.g., 'embedding' → ['em', '##bed', '##ding']). This handles out-of-vocabulary words gracefully and keeps the vocabulary compact.
3. Model inference — The tokenized input is fed through the pre-trained BERT model with a QA head. You get two probability distributions: one for start token positions, one for end token positions.
4. Answer span decoding — You find the token pair (start, end) with the highest joint probability, respecting that start ≤ end. You then map those token indices back to character offsets in the original passage to extract the answer string.
5. Confidence scoring — The probability of the chosen span serves as a confidence score. You can set a threshold to filter out low-confidence answers (e.g., when the model is unsure, fall back to 'no answer').

Handling long contexts: If your passage exceeds 512 tokens, you use a sliding window approach — split the context into overlapping segments, run the model on each, and keep the answer with the highest overall confidence. This is standard practice in production QA systems.

Confidence vs. correctness: High confidence doesn't always mean correct, especially on out-of-domain text. Always evaluate on a held-out set (see Troubleshooting).

Hands-on walkthrough

Let's build a complete, runnable extractive QA system using Hugging Face's transformers library. First, install the dependencies:

pip install transformers torch

Step 1: Load a pre-trained BERT QA model

We'll use distilbert-base-cased-distilled-squad, a distilled version of BERT fine-tuned on the SQuAD dataset. It's fast and accurate enough for most tasks.

from transformers import pipeline

# Quickest path: use the high-level pipeline API
qa_pipeline = pipeline("question-answering", model="distilbert-base-cased-distilled-squad")

context = """
Paris is the capital of France. It is known for its art, fashion, and culture.
The Eiffel Tower, a wrought-iron lattice tower, is a major landmark.
"""
question = "What is the capital of France?"

result = qa_pipeline(question=question, context=context)
print(result)
# OUTPUT: {'score': 0.99, 'start': 0, 'end': 5, 'answer': 'Paris'}

Step 2: Manual pipeline with explicit tokenizer and model

For more control (e.g., custom confidence thresholds, debugging), use the lower-level AutoTokenizer and AutoModelForQuestionAnswering:

from transformers import AutoTokenizer, AutoModelForQuestionAnswering
import torch

model_name = "distilbert-base-cased-distilled-squad"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForQuestionAnswering.from_pretrained(model_name)

question = "Where is the Eiffel Tower?"
context = "The Eiffel Tower is located in Paris, on the Champ de Mars."

inputs = tokenizer(question, context, return_tensors="pt")
with torch.no_grad():
    outputs = model(**inputs)

start_logits = outputs.start_logits[0]
end_logits = outputs.end_logits[0]

# Get the most probable start and end indices
start_idx = torch.argmax(start_logits)
end_idx = torch.argmax(end_logits)

# Convert to answer text
answer_tokens = inputs["input_ids"][0][start_idx:end_idx+1]
answer = tokenizer.decode(answer_tokens, skip_special_tokens=True)
print(answer)  # OUTPUT: Paris

Expected output:

Paris

Step 3: Build a reusable function with confidence filtering

def answer_question(question, context, topk=1):
    inputs = tokenizer(question, context, return_tensors="pt", truncation=True)
    with torch.no_grad():
        outputs = model(**inputs)
    start_probs = torch.softmax(outputs.start_logits, dim=-1)[0]
    end_probs = torch.softmax(outputs.end_logits, dim=-1)[0]
    # naive: best start and end (ignoring constraints)
    start_idx = torch.argmax(start_probs)
    end_idx = torch.argmax(end_probs)
    if start_idx > end_idx:
        return None, 0.0
    answer = tokenizer.decode(inputs["input_ids"][0][start_idx:end_idx+1], skip_special_tokens=True)
    confidence = (start_probs[start_idx] * end_probs[end_idx]).item()
    return answer, confidence

answer, conf = answer_question("Who designed the Eiffel Tower?", "The Eiffel Tower was designed by Gustave Eiffel.")
print(f"Answer: {answer}, Confidence: {conf:.3f}")
# OUTPUT: Answer: Gustave Eiffel, Confidence: 0.983

Compare options / when to choose what

Not all QA systems are equal. Here's a quick comparison of the main approaches you can adopt.

When to choose extractive BERT: - You need deterministic, verifiable answers (e.g., legal, medical, financial). - Your corpus is fixed and clean — answers are present verbatim. - You have limited compute budget.

When to avoid: - Questions require synthesis (e.g., "Summarize the main argument"). - The answer never appears verbatim in the source. - You need to handle multiple documents at once (then use RAG).

Pro tip: For a production-grade system, always combine a retriever (e.g., BM25 or embeddings) with BERT reader — this is the foundation of Retrieval-Augmented Generation (RAG), which you'll likely explore in a later lesson.

Troubleshooting & edge cases

Problem: The model returns a nonsensical or empty answer. - Cause: The answer is not present in the context, or the context is too short. - Fix: Check the context length (minimum 5 tokens). If the context is irrelevant, fine-tune on your domain or use a larger model.

Problem: The answer is a single word but you expected a phrase. - Cause: The model found a high-probability start but a low-probability end, or the end index is before the start. - Fix: Enforce start_idx <= end_idx in your decoding logic (as in the example above). For phrase-level answers, consider using a model fine-tuned on a dataset with longer spans (e.g., Natural Questions).

Problem: The model is slow on CPU. - Cause: Full BERT is heavy. - Fix: Use a distilled or quantized version (e.g., distilbert-base-uncased-distilled-squad) or switch to ONNX Runtime. See the code below for a distilbert example.

Problem: The pipeline fails when the context exceeds 512 tokens. - Cause: BERT's positional embeddings are limited to 512. - Fix: Implement a sliding window: split the context into chunks of, say, 400 tokens with 50-token overlap, run QA on each chunk, and keep the highest-confidence answer.

def long_context_qa(question, long_context, chunk_size=400, overlap=50):
    # tokenize without truncation to get offsets
    tokens = tokenizer.encode(long_context, add_special_tokens=False)
    best_answer = None
    best_score = 0.0
    for start in range(0, len(tokens), chunk_size - overlap):
        chunk_tokens = tokens[start:start+chunk_size]
        chunk_text = tokenizer.decode(chunk_tokens, skip_special_tokens=True)
        answer, score = answer_question(question, chunk_text)
        if answer and score > best_score:
            best_answer, best_score = answer, score
    return best_answer, best_score

Edge case: No answer in context. The model will always output something — even a wrong span. Mitigate by setting a confidence threshold (e.g., 0.5) and returning "Unable to answer" below it.

What you learned & what's next

You now know how to build a question answering system with BERT: you can load a pre-trained extractive QA model, feed it a question and context, and receive a precise answer with a confidence score. You've seen the mental model of BERT as a reader-with-highlighter, understood the tokenization → encoding → span decoding pipeline, and practiced with both the high-level pipeline API and the manual tokenizer/model approach. You've also compared extractive BERT to generative LLMs and RAG, and you're equipped to handle long contexts and low-confidence answers.

Next lesson: In the Python AI track, you'll likely move on to Retrieval-Augmented Generation (RAG) — combining a document retriever with a reader like BERT to answer questions over an entire corpus. That's the natural next step to scale your QA system beyond a single passage.

Now go ahead — extract answers from your own documents and see how much time you save!