Tokenize Your Data for Training

You've cleaned the data, formatted it into instruction-response pairs, and split it into train and validation sets. But if you feed raw strings directly into your model, the training loop will crash with cryptic tensor errors. The missing step: tokenize your data for training. Tokenization converts your text into the numeric IDs your model actually consumes, and if you get it wrong, your fine-tune silently underperforms or fails outright. In this lesson, you'll master tokenization with Hugging Face's tokenizers and transformers libraries, so your dataset is truly model-ready.

The Problem This Lesson Solves

Every transformer model operates on integers, not words. A tokenizer splits your text into subword units—like "finetuning" into ["fin", "etun", "ing"]—and maps each to a unique ID in the model's vocabulary. If your dataset contains raw strings, your training loop will throw errors like TypeError: expected Tensor as element 1 in argument 0, but got str. Even if you manually encode each row, subtle mismatches—wrong padding, inconsistent lengths, or forgetting attention masks—will produce silent degradation.

Consider a real failure: you tokenize your dataset but forget to set return_tensors, so you pass lists of lists to the trainer. The model interprets ragged sequences as a batch, and you get a ValueError deep in the loss computation. Or you use a tokenizer from a different model (e.g., BERT's tokenizer for a GPT-2 model), and the vocabulary IDs are meaningless—your fine-tune learns garbage. This lesson shows you how to avoid these traps.

Core Concept / Mental Model

Think of a tokenizer as a bilingual dictionary between human-readable text and a model's numeric language. The model only understands IDs (e.g., 1623 for "the", 2514 for "token"), so the tokenizer must be consistent between training and inference—same vocabulary, same special tokens, same truncation/padding rules.

A key mental model is the tokenization pipeline:

1. Normalization: Lowercasing, Unicode cleanup, stripping accents (optional).
2. Pre-tokenization: Splitting text on spaces and punctuation to create initial word boundaries.
3. Model: Applying the subword algorithm—WordPiece, BPE, or Unigram—to break words into subword units.
4. Post-processing: Adding special tokens like [CLS], [SEP], or <s> and </s>.
5. Encoding: Converting token list to a tensor of integer IDs, plus an attention mask.

For fine-tuning, you also add two critical steps: truncation to max length and padding to the longest sequence in a batch. Truncation prevents memory blowups; padding ensures all sequences in a batch have the same shape.

Pro tip: The tokenizer you use must match the model's pretraining exactly. If you load huggy/llama-2-7b but use gpt2's tokenizer, every ID will be wrong. Always pair AutoTokenizer with the same model name you're loading.

How It Works Step by Step

Here's the logical sequence to tokenize your dataset correctly:

1. Load a tokenizer that matches your base model.
2. Define your maximum sequence length (e.g., 512 tokens) based on your data's length distribution and hardware limits.
3. Write a tokenization function that takes a batch of text examples, applies truncation and padding, and returns input IDs, attention masks, and possibly labels.
4. Map the function over your dataset using dataset.map(..., batched=True) for efficiency.
5. Convert outputs to PyTorch tensors (or TensorFlow) so the trainer can consume them.
6. Verify the tokenized output by decoding a sample and checking input IDs.

For causal language modeling (e.g., generating text), your labels are the same as input IDs but shifted by one; many trainers handle this internally. For sequence classification, your dataset already has a label column that you don't tokenize.

If your dataset is huge, avoid processing all rows at once; use map with batched=True and possibly num_proc to parallelize. Also, save the tokenized dataset to disk so you don't recompute it on every run.

Hands-On Walkthrough

Let's tokenize a small dataset for fine-tuning a text-generation model. We'll use a sample instruction dataset and the GPT-2 tokenizer (but the same steps apply to Llama or BERT).

Step 1: Load the tokenizer

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("gpt2")
# Add a padding token if missing (GPT-2 has none by default)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

Step 2: Create a toy dataset

from datasets import Dataset

data = {
    "instruction": ["What is 2+2?", "Explain quantum computing"],
    "response": ["4", "Quantum computing uses qubits"]
}
dataset = Dataset.from_dict(data)

Step 3: Write a tokenization function

def tokenize_function(examples):
    # Combine instruction and response into a single text
    texts = [instr + " " + resp for instr, resp in zip(examples["instruction"], examples["response"])]
    # Tokenize with truncation and padding (padding='max_length' is fine for small demos)
    encodings = tokenizer(
        texts,
        truncation=True,
        max_length=64,
        padding="max_length",
        return_tensors="pt"  # returns PyTorch tensors
    )
    return {
        "input_ids": encodings["input_ids"].squeeze(),  # remove batch dim
        "attention_mask": encodings["attention_mask"].squeeze(),
        "labels": encodings["input_ids"].squeeze()  # for causal LM
    }

Step 4: Map over the dataset

tokenized_dataset = dataset.map(tokenize_function, batched=True)
print(tokenized_dataset[0])

Expected output:

{'instruction': 'What is 2+2?', 'response': '4', 'input_ids': tensor([...]), 'attention_mask': tensor([...]), 'labels': tensor([...])}

Notice that input_ids are tensors of length 64 (padded with the EOS token, which is also the pad token).

Step 5: Verify by decoding

sample = tokenized_dataset[0]["input_ids"]
decoded = tokenizer.decode(sample, skip_special_tokens=True)
print(decoded)

Expected output: 'What is 2+2? 4' — the padding tokens are skipped.

Pro tip: In production, use padding=True with a DataCollatorForLanguageModeling to pad dynamically per batch, which is more memory-efficient than padding to max_length for every example.

Compare Options / When to Choose What

If you're fine-tuning a causal LM (like GPT), you typically use padding='longest' with a DataCollatorForLanguageModeling. For sequence classification (BERT-style), a simple DataCollatorWithPadding works.

If your sequences are near uniform length, padding='max_length' is simpler and doesn't hurt. But for realistic datasets with high variance, switch to dynamic padding to save GPU memory.

Troubleshooting & Edge Cases

1. ValueError: Asking to pad but the tokenizer does not have a padding token — Add one, typically the EOS token: python tokenizer.pad_token = tokenizer.eos_token For BERT-style models, they usually have a [PAD] token.

2. Tensor shape mismatch when using return_tensors="pt" inside map — The returned tensors include a batch dimension; use .squeeze() or reshape appropriately, or omit return_tensors and let the data collator handle tensor conversion later.

3. Decoded text contains special tokens like <|endoftext|> — That's expected if you don't skip them in decode. Use skip_special_tokens=True for human-readable inspection.

4. Truncating too aggressively — If max_length is too short, you cut off crucial context. Plot your token length distribution and choose a max length covering the 95th percentile.

5. Forgetting attention masks — When you manually encode, always return attention_mask; otherwise the model may attend to padding tokens, degrading quality.

6. Using the wrong tokenizer — Always verify that the tokenizer ID matches the model ID. A quick check: decode a known word and confirm the split matches the model's known behavior.

What You Learned & What's Next

You've learned the core idea behind tokenization and how it maps text to model-ready IDs. You can now load a tokenizer, define a tokenization function with truncation and padding, and map it over your dataset, while understanding the trade-offs between padding strategies and avoiding common pitfalls. This hands-on skill directly fulfills the objective of converting raw data into tensors for training.

Next in the track, you'll learn how to set up your training loop with Hugging Face's Trainer or a custom PyTorch loop, where you'll feed these tokenized batches into the model. You'll also explore evaluation strategies to measure your fine-tuned model's performance.

Now practice: take your own instruction dataset (or use the sample above) and tokenize it with a different model, like BERT, and observe the difference in token IDs and special tokens. Print a few decoded samples to ensure correctness.