Fine-tune on SQuAD for Q&A

You've mastered general fine-tuning, but when your chatbot, support bot, or search engine needs to extract exact answers from a document, a standard language model falls short. That's the pain this lesson solves: you'll learn to fine-tune a model for question answering on SQuAD — the gold-standard benchmark for extractive QA — so your model can point to the exact span of text that answers a user's question. By the end, you'll have a practical, reusable recipe for building QA systems that work.

The problem this lesson solves

Generic LLMs are brilliant at generating fluent text, but they're not built for precision. Ask a generic model "What is the capital of France?" and it might chat around the answer, hallucinate, or repeat the question. When you need a factual, extractive answer — like pulling the exact sentence from a legal document or a product manual — you need a model that can identify and return the relevant span from the provided context.

This is exactly what SQuAD (Stanford Question Answering Dataset) is designed to train. SQuAD 2.0 adds unanswerable questions, forcing the model to say "I don't know" — a crucial feature for real-world QA systems where the answer might not exist in the source. Without fine-tuning on a dataset like SQuAD, your model will either hallucinate answers or fail to ground them in the provided context. This lesson gives you the exact workflow to adapt a pretrained model to extractive question answering.

Core concept / mental model

Think of fine-tuning for QA as teaching a skilled reader to underline the answer. A pretrained model (like BERT or RoBERTa) already understands language deeply. Fine-tuning on SQuAD adds a small head on top of the base model that outputs two probabilities for every token in the context: the probability that the token is the start of the answer, and the probability that it's the end.

This is a span classification task, not a text generation task. The model reads the question and the context, then selects a contiguous span of tokens that best answers the question. The fine-tuning process adjusts the base model's weights so that this span selection becomes increasingly accurate.

💡 Mental model: A base model is like a dictionary — it knows words and grammar. Fine-tuning on SQuAD turns it into a detective who can pinpoint the exact sentence in a report that answers your question.

Definitions you need

• Extractive QA: The answer must be a substring of the given context — no generation, no paraphrase.
• SQuAD v1.1: Contains ~100,000 question-answer pairs on Wikipedia articles, all answerable.
• SQuAD v2.0: Adds ~50,000 unanswerable questions — the model must predict no answer when appropriate.
• Span head: A classification layer that picks start and end positions for the answer.

How it works step by step

Fine-tuning for question answering follows the same core loop as any Hugging Face fine-tuning, but with a QA-specific twist. Here's the step-by-step logic:

1. Load the dataset: Fetch SQuAD via the datasets library. Start with v1.1 for simplicity, then try v2.0 for robustness.
2. Tokenize with spans: Tokenize the question and context together, but track the character-to-token mapping so you can label the start and end token positions of the answer.
3. Align labels: Convert answer character spans into token indices. For SQuAD v2, also include a "no answer" option (often stored separately).
4. Choose a head: Use AutoModelForQuestionAnswering from Transformers. It adds a span classifier on top of the base encoder.
5. Train: Use the Trainer class with a QA‑aware DataCollator that handles padding and truncation without losing span alignment.
6. Evaluate: Use standard metrics like exact match (EM) and F1 score at the token level.
7. Run inference: Feed a new question+context pair, extract the span with the highest start and end scores.

The critical step is span alignment — if your tokenizer strips characters or adds special tokens, your labels will be off by a few indices, and training silently fails. You'll handle this in the next section.

Hands-on walkthrough

Let's build a complete, runnable example. We'll use a small model (distilbert-base-uncased) and a tiny slice of SQuAD to keep it fast on a CPU. If you have a GPU, bump up the max_samples value.

Setup and data loading

import torch
from transformers import (
    AutoTokenizer,
    AutoModelForQuestionAnswering,
    TrainingArguments,
    Trainer,
    DefaultDataCollator,
)
from datasets import load_dataset

# Load a small slice of SQuAD v1.1 for speed
squad = load_dataset("squad", split="train[:100]").train_test_split(test_size=0.1)
print(squad["train"][0])

Expected output (truncated):

{'id': '5733be284776f41900661182', 'title': 'University_of_Notre_Dame',
 'context': 'The university is the main ...', 
 'question': 'A building that was built in 1882 ...', 
 'answers': {'text': ['the Main Building'], 'answer_start': [1039]}}

Tokenize and align spans

tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")

def preprocess(batch):
    questions = [q.strip() for q in batch["question"]]
    inputs = tokenizer(
        questions,
        batch["context"],
        max_length=384,
        truncation="only_second",
        stride=128,
        return_overflowing_tokens=True,
        return_offsets_mapping=True,
        padding="max_length",
    )

    offset_mapping = inputs.pop("offset_mapping")
    sample_map = inputs.pop("overflow_to_sample_mapping")
    start_positions = []
    end_positions = []

    for i, offset in enumerate(offset_mapping):
        sample_idx = sample_map[i]
        answers = batch["answers"][sample_idx]
        if not answers["answer_start"]:
            start_positions.append(0)
            end_positions.append(0)
            continue

        start_char = answers["answer_start"][0]
        end_char = start_char + len(answers["text"][0])

        # Find the token whose character span contains the answer
        token_start = None
        token_end = None
        for idx, (start, end) in enumerate(offset):
            if start <= start_char and end >= start_char:
                token_start = idx
            if start <= end_char and end >= end_char:
                token_end = idx
        start_positions.append(token_start or 0)
        end_positions.append(token_end or token_start or 0)

    inputs["start_positions"] = start_positions
    inputs["end_positions"] = end_positions
    return inputs

train_dataset = squad["train"].map(preprocess, batched=True, remove_columns=squad["train"].column_names)
eval_dataset = squad["test"].map(preprocess, batched=True, remove_columns=squad["test"].column_names)

⚠️ Pro tip: truncation="only_second" only truncates the context, not the question — crucial to keep the question intact. The stride parameter creates overlapping windows so long contexts don't lose the answer.

Training with the QA head

model = AutoModelForQuestionAnswering.from_pretrained("distilbert-base-uncased")

training_args = TrainingArguments(
    output_dir="./qa-finetuned",
    evaluation_strategy="epoch",
    learning_rate=3e-5,
    per_device_train_batch_size=4,
    per_device_eval_batch_size=4,
    num_train_epochs=2,
    weight_decay=0.01,
    save_total_limit=1,
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    tokenizer=tokenizer,
    data_collator=DefaultDataCollator(),
)

trainer.train()

After training (will take a few minutes on CPU), you can save and load it for inference:

Inference: extract answers from new contexts

def answer_question(question, context, model, tokenizer):
    inputs = tokenizer(question, context, return_tensors="pt", truncation=True, max_length=384)
    with torch.no_grad():
        outputs = model(**inputs)
    start_scores = outputs.start_logits
    end_scores = outputs.end_logits

    start_idx = torch.argmax(start_scores)
    end_idx = torch.argmax(end_scores)
    if start_idx > end_idx:
        return "No answer found"

    answer_tokens = inputs["input_ids"][0][start_idx:end_idx+1]
    return tokenizer.decode(answer_tokens, skip_special_tokens=True)

context = "PythonSkillset offers hands-on LLM fine-tuning tutorials. The platform was created in 2025."
question = "When was PythonSkillset created?"
print(answer_question(question, context, model, tokenizer))
# Expected: "2025" (if trained correctly)

Compare options / when to choose what

You don't always need to fine-tune on SQuAD. Let's compare common approaches to building QA systems:

When to choose fine-tuning on SQuAD: You need deterministic, span‑exact answers (e.g., legal document search, medical record lookup), you have a clear training set or can build one, and you want a small, fast, deployable model.

Variations to explore: - SQuAD v2.0 adds unanswerable questions — critical for production where the answer may not exist. - TyDi QA for multilingual QA (e.g., Arabic, Finnish, Russian) — good if your corpus is multi-language. - SpanBERT or RoBERTa often outperform DistilBERT on SQuAD if you can afford a larger model.

Troubleshooting & edge cases

You'll likely hit a few classic pitfalls. Here's how to diagnose and fix them:

Edge case: overlapping answers — SQuAD v1.1 has a single answer per question, but SQuAD v2 can have none. For handling no-answer, add a separate CLS token head that scores the probability of "no answer" and threshold it during inference.

What you learned & what's next

You now understand fine-tune for question answering on SQuAD end-to-end: you saw why generic LLMs fail at extractive QA, built a mental model of span classification, walked through a complete Hugging Face fine-tuning pipeline, and learned how to align labels, train, and run inference. You also compared fine-tuning with RAG and prompt engineering, so you can choose the right tool for production.

In the next lesson of the LLM Finetuning track, you'll learn how to evaluate your QA model properly using EM and F1 scores, and how to extend the recipe to SQuAD v2.0 with unanswerable questions — preparing you for real-world deployment where accuracy and reliability are non-negotiable.