Preparing Text Data for Finetuning

You’ve spent hours collecting the perfect dataset for your LLM fine-tuning project — hundreds or thousands of instruction-response pairs scrapped from support tickets, docs, or synthetic generation. You are ready to train. Then the first training run fails with a cryptic tokenizer error, or worse, the model trains but completely ignores your fine-tuning signal and just repeats the base model’s default behavior. The culprit? Messy, unformatted data. In this lesson, you’ll learn how to prepare text data for supervised finetuning — the critical step that transforms raw text into a structured, tokenized, and quality-checked dataset your model can actually learn from. This is step six in the LLM Finetuning path, and it builds directly on your previous data collection and prompting work.

The Problem This Lesson Solves

Raw text data is rarely ready for training. It contains inconsistent formatting, extra whitespace, special characters, and mixed languages. More critically, supervised fine-tuning (SFT) requires a specific structure: your model needs to learn to map an input (the instruction or prompt) to an output (the response). Without that structure, training either fails or teaches the model to reproduce patterns you never intended.

Consider these common failure modes:

• Inconsistent prompt formats: Some examples have ### Instruction:, others Q:, and others nothing at all. The model sees chaotic input noise instead of a consistent template.
• Target leakage: The instruction includes the answer, so the model learns to copy rather than reason.
• Truncation disasters: Sequences exceed the model’s context window, and your naive truncation cuts off the response entirely.
• Low-quality pairs: Duplicates, empty responses, or texts with HTML tags degrade the learning signal.

This lesson shows you a repeatable pipeline to solve these issues. You’ll structure, tokenize, and validate your data — so your next training run actually learns the behavior you want.

Core Concept / Mental Model

Think of supervised fine-tuning as teaching a student with flashcards. Each flashcard has a question on one side and the correct answer on the other. Your job is to prepare those flashcards: write the question clearly, write the answer accurately, and make sure every card is the same size so the student can process them efficiently.

Your data preparation pipeline has three core stages:

1. Formatting: Turn each raw example into a structured text block that matches the template your model will see at inference time.
2. Tokenization: Convert formatted text into token IDs (integers) using the model’s tokenizer, with the correct padding and truncation settings.
3. Quality control: Verify that formatted examples are non-empty, contain both instruction and response, and don’t leak answers.

These stages are sequential — you can’t tokenize without formatting, and you shouldn’t train without quality control. The output of this stage is a Dataset object (from Hugging Face datasets) that can be fed directly into a Trainer in the next lessons.

Pro tip: The same pipeline works for any supervised fine-tuning framework (Hugging Face Transformers, Axolotl, LitGPT). It’s not about the tool — it’s about the structure.

How It Works Step by Step

The preparation process follows a strict order. Skipping a step invites hidden errors.

Step 1: Inspect Your Raw Data

Load your raw data and examine a few examples. Use Python to check for nulls, duplicates, and unexpected formats. This step is quick but saves you hours later.

import pandas as pd

df = pd.read_json("data.jsonl", lines=True)
print(df.head(3))
print("Nulls:\n", df.isnull().sum())
print("Duplicates:", df.duplicated().sum())

Expected output: a DataFrame preview plus clear null and duplicate counts.

Step 2: Define a Consistent Template

Choose a template that your base model already understands. For instruction-tuned models like Llama 2, use the standard format:

<s>[INST] {instruction} [/INST] {response} </s>

For other models (like Mistral), use <|user|> and <|assistant|> tokens. Consistency is key — the model learns to map the exact pattern you provide.

Step 3: Format Each Example

Write a format_example function that takes a dict and returns the formatted string. Apply it across the dataset using .map(). Always include a delimiter between instruction and response so the model can separate them during training loss masking.

Step 4: Tokenize

Load the model’s tokenizer and tokenize the formatted strings. Set padding=False (we’ll pad in the collator), truncation=True with max_length equal to your model’s context window (e.g., 2048 for Llama). Add labels — typically a copy of the input IDs, with response position tokens masked to 0 for loss calculation.

Step 5: Run Quality Checks

Decode a few tokenized examples and verify they reconstruct the original text. Check that the response part actually has non-zero labels. Compute sequence length distribution to catch truncation issues.

The result is a clean, tokenized dataset — ready for the Trainer.

Hands-On Walkthrough

Let’s build the full pipeline. You’ll use Hugging Face datasets and transformers.

1. Setup and Load Raw Data

Assume your raw data is a JSONL file with instruction and response fields. (This structure is common from earlier steps in this track.)

from datasets import load_dataset
# Load raw data from a JSONL file
dataset = load_dataset("json", data_files="data.jsonl", split="train")
print(dataset[:2])

Output: a list of two dicts with keys instruction and response.

2. Format the Data

Define a template matching your model. We’ll use the Llama 2 chat format.

# Formatting function
def format_example(example):
    text = f"<s>[INST] {example['instruction']} [/INST] {example['response']} </s>"
    return {"text": text}

formatted_dataset = dataset.map(format_example)
print(formatted_dataset[0]["text"][:200])

Output: the first 200 characters of the formatted string.

3. Tokenize and Build Labels

Load the tokenizer and tokenize the formatted text. Add labels where the response part is preserved (loss is computed) and the instruction part is masked (tokens set to -100). We’ll use a helper to find the response token position.

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-chat-hf")
# For padding, ensure pad_token is set (if not, set to eos_token)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

SFT_TOKEN = "[/INST]"  # token that marks response start

def tokenize_function(example):
    # Tokenize the full text
    tokenized = tokenizer(
        example["text"],
        truncation=True,
        max_length=2048,
        padding=False,
        return_tensors="pt"
    )
    input_ids = tokenized["input_ids"][0]
    # Find the response start index (the token after [/INST])
    # Search for the string "[/INST]" in the decoded text
    response_start_char = example["text"].find(SFT_TOKEN) + len(SFT_TOKEN)
    response_start_token = len(tokenizer.encode(example["text"][:response_start_char])) - 1
    labels = input_ids.clone()
    labels[:response_start_token] = -100  # mask instruction
    return {"input_ids": input_ids, "attention_mask": tokenized["attention_mask"][0], "labels": labels}

tokenized_dataset = formatted_dataset.map(tokenize_function, remove_columns=["text","instruction","response"])
print(tokenized_dataset[0])

Output: a dict with input_ids, attention_mask, and labels — every example now a tuple of token IDs.

Pro tip: Use padding=True inside the collator (like DataCollatorForLanguageModeling) to batch variable-length sequences, not during tokenization — this saves memory and speeds up training.

4. Verify Quality

Decode a sample and check labels are correct.

# Decode the first example
text = tokenizer.decode(tokenized_dataset[0]["input_ids"])
print(text[:300])
print("Labels non-zero count:", (tokenized_dataset[0]["labels"] != -100).sum().item())

Expected output: the original text (with special tokens) and a non-zero count equal to the number of response tokens.

Compare Options / When to Choose What

Not all tokenization strategies are equal. Here’s how to choose:

For most fine-tuning, dynamic padding with truncation at the model’s context window is the sweet spot. If your responses are typically short, filter out the top 1% longest examples instead of truncating them.

Variation 1: Use the Hugging Face tokenizers library’s post_processors (like TemplateProcessing) to add special tokens during tokenization automatically — good for reproducible pipelines.

Variation 2: Use a chat template built into your tokenizer (tokenizer.apply_chat_template) for models like Mistral or Llama 3, which simplifies formatting but requires you to know the exact template.

Variation 3: For extremely large datasets (>1M examples), consider pre-tokenizing and saving to Arrow format (save_to_disk) to avoid re-tokenizing every time you tweak hyperparameters.

Troubleshooting & Edge Cases

• Tokenizer error: Token indices sequence length is longer than the specified maximum sequence length — You set max_length too low. Increase it or filter out longer examples.
• Model outputs nonsense after training — Your labels mask the response incorrectly. Check that labels are -100 on the instruction part and non-zero on the response.
• All labels are -100 — Your response_start_token calculation is off. Print the tokenizer.encode of the prefix and verify the index.
• Duplicates in data cause overfitting — Use dataset = dataset.filter(lambda x: x["text"] not in seen) for production data.
• Special tokens like <s> appear in training but not inference — Always apply the same formatting at inference (use the same prompt template).
• Empty instructions after formatting — Filter out examples where len(ins) == 0.

What You Learned & What's Next

You now understand how to prepare text data for supervised finetuning. You can format raw text into consistent templates, tokenize with proper truncation and padding, and label-mask instruction parts so the model learns only from responses. You’ve verified your dataset’s integrity with simple decode checks. This is the foundation for the next lesson in the LLM Finetuning track, where you’ll dive into parameter-efficient fine-tuning with LoRA and QLoRA. Armed with a clean, tokenized dataset, you’re ready to train a custom model without wasting GPU hours on noisy data.

Key points to remember: Always inspect your raw data, use a consistent template, mask instruction tokens in labels, and validate your tokenized output before training.

Ready to fine-tune? Move to the next lesson and put this dataset to work.