How to Truncate Text to a Token Budget in Python

Truncate a string to a maximum token budget for LLM context using the tiktoken library and OpenAI's tokenizer.

Easy Python 3.9+ Aug 9, 2026 AI & LLM integration patterns 16 views 0 copies

tiktoken llm tokens truncation

Requires third-party packages — install first

pip install tiktoken

Python code

15 lines

Python 3.9+

import tiktoken

def truncate_to_token_budget(text, max_tokens, model="gpt-3.5-turbo"):
    enc = tiktoken.encoding_for_model(model)
    tokens = enc.encode(text)
    if len(tokens) <= max_tokens:
        return text
    truncated_tokens = tokens[:max_tokens]
    return enc.decode(truncated_tokens)

if __name__ == "__main__":
    sample_text = "This is a long sentence. " * 20
    result = truncate_to_token_budget(sample_text, max_tokens=10)
    print(result)
    print(f"Token count: {len(tiktoken.encoding_for_model('gpt-3.5-turbo').encode(result))}")

Output

stdout

This is a long sentence. This is a long
Token count: 10

How it works

The tiktoken.encoding_for_model returns the tokenizer appropriate for the given model, and enc.encode splits the text into tokens. If the token count exceeds the budget, we slice the token list to the maximum allowed and decode back to text. This preserves word boundaries better than simple character slicing, since the tokenizer is aware of the model's vocabulary. The result is a truncated string that fits within the token limit, ready to be sent to the model.

Common mistakes

Using `max_tokens` as a character limit instead of token count
Forgetting that `tiktoken` encoding is model-specific; using the wrong model changes tokenization
Not accounting for tokens added by prompt templates or system messages when setting the budget

Variations

Instead of slicing tokens, truncate by character count as a cheaper approximation when tokenizer is not available
Use `cl100k_base` encoding directly with `tiktoken.get_encoding` when the model is not known

Real-world use cases

Preprocessing user input before sending to a language model to stay within context window limits.
Truncating logs or long documents for summarization where the model accepts a fixed maximum input size.
Building a chat history manager that drops oldest messages when the conversation exceeds the token budget.

How to Truncate Text to a Token Budget in Python

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from AI & LLM integration patterns

Tutorials

Quizzes

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from AI & LLM integration patterns

Keep learning

Tutorials

Quizzes