How to Estimate Token Count in Python

Estimates tokens in a text string using a whitespace and punctuation heuristic without external libraries.

Easy Python 3.9+ Aug 9, 2026 AI & LLM integration patterns 13 views 0 copies

token-count llm heuristic ai text-processing

Python code

17 lines

Python 3.9+

def estimate_tokens(text: str) -> int:
    """Estimate token count using whitespace and punctuation heuristics."""
    if not text:
        return 0

    words = text.split()
    total_punctuation = sum(1 for char in text if char in ".,!?;:")
    special_tokens = sum(1 for char in text if char in "\n\t")

    # Rough estimate: ~1.3 tokens per word + punctuation + special characters
    estimate = int(len(words) * 1.3) + total_punctuation + special_tokens
    return max(estimate, 1)


if __name__ == "__main__":
    sample_text = "Hello world! This is a test.\nIt has multiple lines."
    print(f"Token estimate: {estimate_tokens(sample_text)}")

Output

stdout

Token estimate: 16

How it works

The heuristic splits text into words and counts punctuation and special characters. len(words) * 1.3 approximates tokens per word, with punctuation and newlines added. This provides a fast, rough estimate useful for budgeting LLM API costs. It works best for English prose and may undercount for code or other languages.

Common mistakes

Ignoring punctuation-only tokens like apostrophes or hyphenated words
Assuming the estimate matches exact tokenizer output (e.g., tiktoken)
Forgetting to handle empty strings, which returns 0

Variations

Use `tiktoken.get_encoding('cl100k_base')` for exact OpenAI token counts
Adjusted multiplier for non-English texts (e.g., 2.5 for Chinese/Spanish)

Real-world use cases

Estimating API costs before sending large prompts to an LLM endpoint.
Budgeting token limits when batching multiple user messages into one request.
Serving as a quick sanity check before a costly embedding or completion call.

How to Estimate Token Count in Python

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from AI & LLM integration patterns

Tutorials

Quizzes

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from AI & LLM integration patterns

Keep learning

Tutorials

Quizzes