How to Compute a Mock BLEU Score with n-gram Overlap in Python

Evaluate text similarity with a simplified BLEU score using word-level n-gram precision and a brevity penalty.

Easy Python 3.9+ Aug 9, 2026 AI & LLM integration patterns 12 views 0 copies

Python code

36 lines
Python 3.9+
from collections import Counter

def bleu_score(reference, candidate, n=2):
    """
    Compute a simplified BLEU score with n-gram precision and brevity penalty.
    Mock demo using word-level n-grams.
    """
    ref_tokens = reference.lower().split()
    cand_tokens = candidate.lower().split()
    
    # Compute n-gram precision
    ref_ngrams = Counter(
        tuple(ref_tokens[i:i+n]) 
        for i in range(len(ref_tokens) - n + 1)
    )
    cand_ngrams = Counter(
        tuple(cand_tokens[i:i+n]) 
        for i in range(len(cand_tokens) - n + 1)
    )
    
    overlap = sum((cand_ngrams & ref_ngrams).values())
    total_cand_ngrams = max(sum(cand_ngrams.values()), 1)
    precision = overlap / total_cand_ngrams
    
    # Brevity penalty
    ref_len = len(ref_tokens)
    cand_len = len(cand_tokens)
    brevity_penalty = min(1.0, ref_len / cand_len) if cand_len > 0 else 0.0
    
    # Simplified BLEU (1-gram precision * brevity penalty)
    return precision * brevity_penalty

if __name__ == "__main__":
    reference_text = "the cat sat on the mat"
    candidate_text = "the cat sat on mat"
    print(f"BLEU score: {bleu_score(reference_text, candidate_text, n=2):.3f}")

Output

stdout
BLEU score: 0.500

How it works

This demo implements a stripped-down BLEU metric: it tokenizes both texts, builds n-gram counters, and computes precision as the overlap ratio. The brevity penalty adjusts for candidates shorter than the reference, discouraging overly terse outputs. Using Counter & Counter computes the minimum count per n-gram, exactly measuring matches. This simplified version ignores the geometric mean across multiple n-gram lengths and length-based smoothing, but it is enough to demonstrate the concept.

Common mistakes

  • Using `json.loads` when you meant `json.load` for file objects
  • Forgetting to lowercase or tokenize text consistently, skewing n-gram matches
  • Dividing by zero when the candidate has no n-grams — guard with `max(..., 1)`
  • Not applying a brevity penalty, so short guesses score too high

Variations

  1. Use `nltk.translate.bleu_score.sentence_bleu` for a full BLEU implementation
  2. Extend to multiple n-gram sizes (1..4) and average the log precision

Real-world use cases

  • Quickly validating machine translation quality before deploying a model to production.
  • A regression test for text generation, ensuring summaries or captions stay close to expected output.
  • Comparing LLM responses against a reference answer to score RAG pipeline accuracy.

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from AI & LLM integration patterns

Related tutorials and quizzes for this topic.