Chat Formatting with Tokenizers

Fine-tuning a chat model without a consistent chat format is the fastest way to produce a model that confuses your users — it might answer your system prompt, ignore conversation history, or spew out hidden tokens. The root cause isn't your training data or hyperparameters; it's the way you tokenize conversations. When you learn to use tokenizers for chat formatting via apply_chat_template, you align your data with the model's pretraining format, which is the single most reliable lever for predictable, high-quality chat responses.

The problem this lesson solves

Pretrained chat models are tuned on a specific chat template — a canonical sequence of special tokens that mark the system message, user turns, and assistant replies. If you feed your own fine-tuning data in a plain or arbitrary format, you're teaching the model to ignore these markers during training, and at inference it will generate gibberish like ### User: or <|im_start|>user on its own.

Consider what happens when your data is not formatted consistently:

• The model cannot reliably distinguish who said what, so it may answer for the user instead of the assistant.
• System instructions get ignored because the model never learned that special system tokens indicate directives.
• Multi-turn context degrades; the model forgets earlier turns or repeats them.
• Your training loss looks fine, but your evaluation metrics (like open-ended human preference) are terrible.

The exact same fine-tuning run can produce a useless chatbot or a polished one — the difference is the token-level formatting applied before training.

Core concept / mental model

Think of a tokenizer's chat template as the grammar of a conversation. Just as a programming language has syntax rules, a chat model expects conversations to be expressed with a precise sequence of special tokens. The tokenizer's job is to turn a list of messages (system, user, assistant) into that exact token sequence.

A chat template is a Jinja2 template stored in the tokenizer's chat_template attribute. It defines:

• Special tokens like <|im_start|> and <|im_end|> (for ChatML) or [/INST] (for Llama 2).
• Structure — when to emit system, user, assistant roles and how to separate turns.
• Placeholders for message content and role.

Key definitions:

• Tokenization — converting raw text into integer IDs.
• Chat template — a function that maps a list of messages to a tokenized (or string) representation.
• apply_chat_template — the method on PreTrainedTokenizer that performs this transformation.

Mental diagram:

Messages (list of dicts)
        |
        v
[apply_chat_template] --> string with special tokens
        |
        v
[tokenizer(...)]     --> input_ids (integers)
        |
        v
( training / generation )

How it works step by step

1. Load a tokenizer that has a chat template, typically from the same model you're fine-tuning (or a compatible one).
2. Prepare your messages as a list of dictionaries, each with a role ("system", "user", "assistant") and content (the text).
3. Call apply_chat_template on that list, with tokenize=False to see the formatted string or tokenize=True to get input IDs.
4. Ensure return_tensors (e.g., "pt") to get PyTorch tensors ready for the training loop.
5. Apply the template to every conversation in your dataset consistently, both at training and at inference.

Always confirm the template matches the model's architecture. Using a Llama-2 template on a Mistral model will corrupt the conversation.

Hands-on walkthrough

Let's load a chat model tokenizer and apply the template. We'll use the Hugging Face transformers library (v4.38+).

Example 1: Basic chat formatting

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("HuggingFaceH4/zephyr-7b-beta")

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Tell me a joke about Python."},
]

formatted = tokenizer.apply_chat_template(messages, tokenize=False)
print(formatted)

Output (truncated):

<|system|>
You are a helpful assistant.</s>
<|user|>
Tell me a joke about Python.</s>
<|assistant|>

Notice that the template automatically appended the <|assistant|> tag, which is critical because it tells the model it's time to generate.

Example 2: Tokenizing and preparing for training

from transformers import AutoTokenizer

messages = [
    {"role": "system", "content": "You are a coding assistant."},
    {"role": "user", "content": "Write a Python function to sort a list."},
    {"role": "assistant", "content": "Here is a simple sort:\n\ndef sort_list(lst):\n    return sorted(lst)"},
    {"role": "user", "content": "Can you make it a one-liner?"},
]

inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    return_tensors="pt",
)

print(inputs['input_ids'].shape)  # torch.Size([1, N])
print(tokenizer.decode(inputs['input_ids'][0]))

The output is a tensor of token IDs representing the entire formatted conversation. When training, you'll also need to mask the assistant-only labels.

Example 3: Training data pipeline with datasets

from datasets import Dataset
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.2")

def format_conversation(example):
    messages = example["messages"]  # assume dataset has a 'messages' field
    text = tokenizer.apply_chat_template(messages, tokenize=False)
    return {"text": text}

dataset = dataset.map(format_conversation)
# Now tokenize and pad to max_length or pack sequences.

Compare options / when to choose what

Variations:

• Some tokenizers offer add_generation_prompt=True to append the assistant tag for inference.
• You can override tokenizer.chat_template to a custom Jinja2 template if needed.
• tokenizer.apply_chat_template accepts tokenize=False for string output and return_dict=True to get a full batch dictionary.

Troubleshooting & edge cases

1. Tokenizer has no chat_template?

if tokenizer.chat_template is None:
    # Load a compatible tokenizer that has one, or set it manually.

2. Messages missing a role key?

The template expects role and content. A KeyError means your data structure is wrong. Validate your dataset.

3. Inconsistent tokenization between training and inference?

Always use the same apply_chat_template call, including add_generation_prompt for inference. If you train with the assistant tag but generate without it, the model sees a different starting context.

4. Padding to the left vs. right?

For causal LMs, pad on the left for generation. For training, right-padding is common but you must mask padding tokens in the loss.

5. Special tokens not in the tokenizer?

If your template uses custom tokens like <|im_start|>, ensure they are added to the tokenizer (tokenizer.add_special_tokens). If you add new tokens, you must resize the model's embedding layer.

What you learned & what's next

You now know why chat formatting matters, how apply_chat_template works under the hood, and how to integrate it into tokenization pipelines for fine-tuning. You can format any dataset of messages and produce ready-to-train token IDs.

Next: In the next lesson, you'll learn how to mask loss on the prompt tokens so the model only learns from assistant responses — a natural follow-up to ensure your fine-tune actually learns to answer.

Continue your LLM Finetuning track to keep building expertise.