LLM Truncation Basics

You've got a dataset of long product reviews, chat logs, or legal documents, and you're ready to fine-tune your LLM. You load the data, tokenize it, and hit a wall of errors and memory crashes. This is the exact moment you need to master truncation — the art of cutting tokenized sequences down to a size your model can actually digest. Without it, your training loop will be slow, unstable, and ultimately fail; with it, you'll keep your runs fast, your memory footprint small, and your model's performance high.

The problem this lesson solves

Every transformer model has a maximum context window — a hard limit on how many tokens it can process in a single forward pass. For models like BERT it's typically 512 tokens; for GPT-2 it's 1024; for LLaMA-2 it's 4096. When your input sequence exceeds that limit, you have three choices: crash, silently truncate, or carefully handle the overflow. Most fine-tuning frameworks will crash with an ugly out-of-memory error if you don't handle truncation upfront.

The pain is real:

• OOM (Out-of-Memory) errors that kill your training job mid-epoch.
• Silently dropped data when the tokenizer truncates by default but you didn't realize it.
• Terrible model quality because you truncated the wrong end of the sequence, losing the answer or the key context.
• Massive wasted compute when your sequences are 10x longer than they need to be.

The solution is simple: learn to handle long sequences with truncation deliberately. You decide what to keep, what to cut, and where the cut happens — instead of letting the framework do something random and destructive.

Core concept / mental model

Think of tokenization as packing a suitcase and truncation as deciding what to leave at home. Your model has a fixed-size suitcase (the context window). You have a mountain of clothes (your text). Truncation is the packing strategy you choose.

Here's the key mental model — a tokenizer's truncation argument can behave in one of three ways:

1. No truncation (default in many low-level APIs): If your input is longer than the max length, it gets passed as-is — and the model blows up.
2. True or 'longest_first' (the best default): The tokenizer automatically trims the sequence to fit the max length.
3. 'only_first' or 'only_second': For paired inputs (like question-answer pairs or instruction-response pairs), it truncates only one of the two texts.

The critical distinction is where the cut happens. For causal language models (like GPT), you generally want to keep the beginning of a document and drop the tail. For sequence classification (like BERT), you might want to keep the end if the label is at the bottom of the text. The truncation parameter gives you this control.

Here's the flow you'll internalize:

Raw text -> Tokenize (with truncation=True, max_length=N) -> Padded batch -> Model forward pass

How it works step by step

When you call a tokenizer with truncation enabled, here's what happens under the hood, step by step:

1. Tokenization: The raw text is split into tokens (subwords or words). This produces a list of token IDs.
2. Length check: The tokenizer compares the length of the token list against max_length.
3. Decision (depends on truncation value): - If truncation=True, it keeps the first max_length tokens (by default) and discards the rest. - If truncation='only_first', for a pair of sequences, it truncates only the first one to fit. - If truncation='longest_first', it removes one token from whichever sequence is currently longest, iteratively, until both fit.
4. Padding: After truncation, the tokenizer pads shorter sequences to the max length in the batch (using padding=True) so you get a consistent tensor shape.
5. Attention mask: It marks real tokens with 1 and padded tokens with 0.

The cause-and-effect chain is critical: max_length defines the ceiling. truncation decides if you cut. padding fills in the gaps. Get any of these wrong, and your training is corrupted — either with crashes or with non-informative padded tokens.

Pro tip: Always set truncation=True explicitly. Don't rely on defaults. In the Hugging Face transformers library, the behavior changed across versions, and silent bugs are the worst kind.

Hands-on walkthrough

Let's put this into practice with a concrete example. We'll use the Hugging Face transformers library with a small dataset of long product reviews.

Step 1: Set up the tokenizer

from transformers import AutoTokenizer

# Use a small BERT model for the demo
model_name = "bert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Our "long" sample text — way more than 512 tokens when tokenized
long_text = "This product changed my life. " * 200  # ~2000 words

print(f"Raw text length: {len(long_text.split())} words")

Expected output:

Raw text length: 1200 words

Step 2: Tokenize with truncation

# Tokenize WITH truncation to fit the model's default max length (512 for BERT)
encoded = tokenizer(
    long_text,
    truncation=True,
    max_length=512,
    padding="max_length",
    return_tensors="pt"
)

print(f"Input IDs shape: {encoded['input_ids'].shape}")
print(f"Attention mask shape: {encoded['attention_mask'].shape}")
print(f"Number of actual tokens: {encoded['attention_mask'].sum().item()}")

Expected output:

Input IDs shape: torch.Size([1, 512])
Attention mask shape: torch.Size([1, 512])
Number of actual tokens: 512

Step 3: See the truncation in action

# Compare against NOT truncating (this would crash in the model)
encoded_no_trunc = tokenizer(long_text, return_tensors="pt")
print(f"Without truncation: {encoded_no_trunc['input_ids'].shape[1]} tokens")
print(f"With truncation: {encoded['input_ids'].shape[1]} tokens")

# Verify truncation kept the beginning
full_text = tokenizer.decode(encoded["input_ids"][0], skip_special_tokens=True)
print(f"\nDecoded truncation begins with: {full_text[:80]}...")

Expected output:

Without truncation: 4200 tokens
With truncation: 512 tokens

Decoded truncation begins with: This product changed my life. This product changed my life. This product changed my life....

Step 4: Batch tokenization (the real-world use case)

from datasets import Dataset

# Simulate a real dataset
data = {
    "text": [
        "Short review",
        "A " * 1000,  # 1000 tokens
        "Another " * 300,  # 300 tokens
        "Mixed length text " * 700,  # 700 tokens
    ]
}
dataset = Dataset.from_dict(data)

# Tokenize the whole dataset with truncation
def tokenize_function(examples):
    return tokenizer(
        examples["text"],
        truncation=True,
        max_length=512,
        padding="max_length"
    )

tokenized_dataset = dataset.map(tokenize_function, batched=True)
print(tokenized_dataset)
print(f"All sequences are exactly 512 tokens: {tokenized_dataset['input_ids'][0]}")

Expected output:

Dataset({features: ['text', 'input_ids', 'token_type_ids', 'attention_mask'], num_rows: 4})
All sequences are exactly 512 tokens: [list of 512 ints]

Step 5: Control which side gets truncated

# With 'only_first', we can preserve the second sequence in a pair
question = "What are the battery specs? " * 300  # 600 tokens
answer = "The battery lasts 20 hours."

encoded_pair = tokenizer(
    text=question,
    text_pair=answer,
    truncation="only_first",
    max_length=512,
    padding="max_length",
    return_tensors="pt"
)

# The answer should be intact at the end
decoded = tokenizer.decode(encoded_pair["input_ids"][0], skip_special_tokens=True)
print(f"Ends with (should contain the answer): ...{decoded[-50:]}")

Expected output:

Ends with (should contain the answer): ...The battery lasts 20 hours.

Compare options / when to choose what

When to choose what:

• Most fine-tuning: Use truncation=True, max_length=model_max_length. Simple and effective.
• Instruction fine-tuning (LoRA, QLoRA): Use truncation='only_first' if the instruction is long and the response is short. Preserve the response!
• Document classification: Use truncation=True if the label is in the first few sentences; otherwise, consider truncating the tail and keeping the head — which is the default anyway.
• Long document summarization: Don't rely on truncation alone. Use a sliding window approach (chunking) instead of a single hard cut.

Pro tip: For a quick sanity check, log the attention_mask.sum() per sample. If your average is consistently near max_length, you have significant truncation happening — and you should consider chunking instead.

Troubleshooting & edge cases

Error: Token indices sequence length is longer than the specified maximum sequence length

• Cause: You didn't pass truncation=True to the tokenizer.
• Fix: Add truncation=True, max_length=512 (or your model's limit) to the tokenizer call.

Error: Out-of-memory (OOM) during training

• Cause: Your batch contains one very long sequence that pushes you over the memory limit, even if you truncated. This happens when padding="max_length" and you have long sequences that were properly truncated but the batch is too large.
• Fix: Reduce batch size, or use padding=True (dynamic padding) instead of padding="max_length" to pad to the longest in the batch, not a fixed length.

Wrong behavior: The answer (or key info) is missing from the output after truncation

• Cause: You used truncation=True on a pair, and the answer was at the end of the long text, which got cut off.
• Fix: Use truncation='only_first' or reorder your inputs so the critical text is in the first sequence.

Edge case: Sequences shorter than max_length get padded with [PAD] tokens

• This is normal and expected. The attention mask tells the model to ignore them. But if you have tons of padding, you waste compute. Use padding=True (dynamic) to pad to the longest in the batch.

Edge case: Tokenizer adds special tokens after truncation

• BERT adds [CLS] and [SEP]; GPT adds <|endoftext|>. These count toward your max_length. If you set max_length=512, you might only get 510 real content tokens. Account for this by adding a small buffer: max_length=model_max_length - 2.

What you learned & what's next

You now know how to handle long sequences with truncation. You've learned that truncation is not just a "cutting" operation — it's a strategic decision about what information to preserve. You've practiced setting max_length, choosing truncation strategies like 'only_first' and 'longest_first', and debugging common tokenization errors.

The core takeaways to remember:

• Always set truncation=True explicitly in your tokenizer calls.
• Choose the right truncation strategy based on where the critical information sits in your text.
• Mind the special tokens that count toward your max length.
• For very long texts, truncation alone is a lossy shortcut — consider chunking instead.

You've mastered two of the key learning objectives: explaining the core idea behind truncation and completing a practical exercise. With this foundation, you're ready to move to the next lesson in the LLM Finetuning track, where you'll tackle padding and attention masks in detail — the perfect companion to truncation that ensures your batched training runs are both correct and efficient.