Attention Masks & Padding

You've spent hours curating the perfect dataset and carefully formatting your prompts, but your model still produces gibberish or worse — silently ignores entire parts of your input. The culprit is often not your data or your model, but a subtle detail in how you handle sequences of different lengths: padding and attention masks. When you batch examples of varying lengths, you must pad them to a uniform size, and without a mask telling the model to ignore those padding tokens, it will attend to empty space, polluting the learned representations and degrading performance. This lesson demystifies attention masks and padding — why they exist, how they work, and how to use them correctly in your LLM finetuning pipeline.

The problem this lesson solves

Imagine you're building a text classification model for customer support tickets. Your training data has queries like "refund" (7 characters) and "How do I return a defective item?" (32 characters). When you put these in a batch, your tokenizer produces sequences of different lengths: [4 tokens] and [11 tokens]. Most deep learning frameworks expect batches as rectangular tensors — every sample must have the same number of tokens. Padding fills shorter sequences with a special token (often [PAD]) to make them all the same length. But here's the twist: the model doesn't inherently know which tokens are real and which are padding. Without an attention mask — a binary tensor that tells the attention mechanism which positions to ignore — the model will treat padding as real data. It will compute attention over those meaningless tokens, skewing the learned representations, and during finetuning, the loss will be calculated over padding too, wasting computation and potentially corrupting gradients. This is a silent killer: your model might still train, but performance will be noticeably worse, and you'll wonder why.

Core concept / mental model

Think of a teacher grading essays of different lengths. To make the stack uniform, you ask students to fill empty space with a placeholder word like "BLANK". The teacher must know to ignore "BLANK" when grading — otherwise, counting words that are just placeholders would inflate the essay length. In LLM finetuning, padding is the "BLANK" filler, and the attention mask is the teacher's instruction: "Ignore the BLANKs."

Let's define the two key terms:

• Padding: Adding special tokens (e.g., [PAD]) to the end (or often beginning) of sequences to make them equal length within a batch. It's purely a computational necessity.
• Attention mask: A tensor of the same shape as the input IDs, typically containing 1 for real tokens and 0 for padding tokens. During the attention computation, these zeros effectively mask out the padding, preventing the model from attending to them.

In Hugging Face's Transformers library, these are passed to the model as attention_mask. The model internally uses this mask to set the attention scores of padding positions to -infinity before the softmax, so they contribute zero attention weight.

How it works step by step

Let's trace the flow from raw text to the model's forward pass:

1. Tokenize each sample individually. The tokenizer converts text to token IDs, yielding sequences of varying lengths.
2. Pad to the batch's maximum length. For each batch, find the longest sequence and pad all shorter ones with [PAD] tokens. The tokenizer's padding=True in your tokenizer(...) call handles this automatically when you pass a list of texts.
3. Generate the attention mask. The tokenizer also produces attention_mask when padding=True. This mask has 1 at real token positions and 0 at padding positions.
4. Pass both to the model. During model(**batch), the mask is used inside the transformer layers. For each attention head, the query attends to keys; positions with mask value 0 are excluded (attention score set to -inf).
5. Loss calculation skips padding. When computing the loss for language modeling or sequence classification, the model (or your loss function) uses the mask to ignore padding tokens, so only real labels contribute.

Here's a minimal tokenization example to see the mechanics:

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

texts = ["I love Python.", "Finetuning LLMs is exciting and rewarding!"]

encoded = tokenizer(texts, padding=True, truncation=True, return_tensors="pt")

print("Input IDs:")
print(encoded["input_ids"])
print("\nAttention Mask:")
print(encoded["attention_mask"])

Expected output (simplified):

Input IDs:
tensor([[ 101, 1045, 2293, 3390, 1012, 102, 0, 0],
        [ 101, 6166, 22013, 6278, 2003, 7697, 1998, 6629]])

Attention Mask:
tensor([[1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1]])

Notice that the mask aligns with the input IDs: 0 indicates the [PAD] token (ID 0 in this case).

Hands-on walkthrough

Now let's see how this works in a real finetuning loop. We'll use a small BERT model for sentiment classification with a custom training loop from Hugging Face.

Step 1: Set up and prepare data

from transformers import AutoTokenizer, AutoModelForSequenceClassification
from torch.utils.data import Dataset

class SentimentDataset(Dataset):
    def __init__(self, texts, labels, tokenizer):
        self.encodings = tokenizer(texts, truncation=True, padding=True, return_tensors="pt")
        self.labels = labels

    def __getitem__(self, idx):
        item = {key: val[idx] for key, val in self.encodings.items()}
        item["labels"] = self.labels[idx]
        return item

    def __len__(self):
        return len(self.labels)

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased", num_labels=2)

texts = ["This product is amazing!", "Terrible experience, never again.", "It works fine."]
labels = [1, 0, 1]
train_dataset = SentimentDataset(texts, labels, tokenizer)

Step 2: Run the forward pass and compute loss

import torch

batch = next(iter(torch.utils.data.DataLoader(train_dataset, batch_size=2, shuffle=True)))
outputs = model(**batch)
print("Loss:", outputs.loss.item())
print("Logits shape:", outputs.logits.shape)

This works because the attention_mask and input_ids are both passed automatically as part of the batch dictionary. The model uses the mask to ignore padding in both attention and loss.

Step 3: Examine the effect of a mask on loss

To see the difference, let's manually compute the loss with and without the mask:

from torch.nn import CrossEntropyLoss

# Without mask: we force logits at padding positions
logits = outputs.logits
labels = batch["labels"]
loss_fct = CrossEntropyLoss()
loss_without_mask = loss_fct(logits, labels)

# With mask: we filter out padding positions before loss
active_logits = logits[batch["attention_mask"] == 1]
active_labels = labels[batch["attention_mask"] == 1]
loss_with_mask = loss_fct(active_logits, active_labels)

print(f"Loss without mask: {loss_without_mask.item():.4f}")
print(f"Loss with mask (correct): {loss_with_mask.item():.4f}")

The loss with the mask is the correct one; the loss without the mask is artificially skewed because it includes padding tokens' logits.

Compare options / when to choose what

There are several strategies for handling variable-length sequences. Understanding them helps you set the right padding side and truncation strategy.

Pro tip: For decoder-only models (like GPT), always use left padding if you plan to generate text. Right padding would place [PAD] at the end where the model expects to generate the next token.

Troubleshooting & edge cases

• Issue: Model predicts [PAD] tokens. If you use right padding with a generation model, the model might generate padding tokens and continue to infinite loops. Fix: use left padding and an appropriate tokenizer setting.
• Issue: Loss is NaN during training. This can happen if your padding token ID is not set for the model, or if you didn't pass attention_mask and the model computes loss over padding. Ensure tokenizer.pad_token is defined (if not, set tokenizer.add_special_tokens({'pad_token': '[PAD]'}) and resize the model embeddings).
• Issue: Mask shape mismatch. In custom loss functions, be careful with index selection. Use torch.where or boolean masking as shown.
• Edge case: Truncation and padding interaction. If a sequence is longer than the model's max_len, you must truncate. The tokenizer handles this, but ensure truncation=True to avoid runtime errors.

Common mistakes:

• Forgetting to pass attention_mask to the model — silently degrades performance.
• Using right padding for generative models — causes hallucinations and bad samples.
• Not setting padding=True in the tokenizer — leads to shape mismatches or unintended truncation.
• Mixing up padding sides — using default right padding for decoder-only models.

What you learned & what's next

You now understand the core mechanics of attention masks and padding in LLM finetuning: why padding is needed for batching, how attention masks tell the model which tokens to ignore, and how to use them in Hugging Face pipelines. You've seen how padding side matters for different model architectures and how dynamic padding improves efficiency. You can now confidently prepare batches that don't poison your model with meaningless padding.

What's next? In the next lesson, you'll learn how to handle long documents and sequence length limits — a natural extension of padding and truncation. You'll explore striding and chunking strategies to feed long text into models with fixed context windows.

Practice recap: Take a dataset of mixed-length texts, tokenize with padding and masks, and run a forward pass with a language model. Try toggling left vs. right padding and observe how the attention pattern changes for a generative model.

Key takeaways:

• Padding makes batches rectangular; attention masks tell the model to ignore padding tokens.
• Always pass attention_mask to the model — it's crucial for correct loss and attention.
• Choose padding side based on model type: right for bidirectional, left for causal/generation.
• Dynamic padding via DataCollatorWithPadding saves compute and is the standard for finetuning.
• Verify your tokenizer has a pad_token; otherwise the mask is meaningless.
• Use the mask in custom loss functions to exclude padding positions.

Real-world use cases:

• Finetuning a chatbot on customer support conversations with varied message lengths — attention masks keep the model focused on real content.
• Training a sentiment classifier on product reviews that range from one word to paragraphs — dynamic padding makes batching efficient.
• Finetuning a code generation model on functions of different sizes — left padding ensures the model can generate without hitting padding artifacts.