How to Compute a Mock BLEU Score with n-gram Overlap in Python
Evaluate text similarity with a simplified BLEU score using word-level n-gram precision and a brevity penalty.
Python code
36 linesfrom collections import Counter
def bleu_score(reference, candidate, n=2):
"""
Compute a simplified BLEU score with n-gram precision and brevity penalty.
Mock demo using word-level n-grams.
"""
ref_tokens = reference.lower().split()
cand_tokens = candidate.lower().split()
# Compute n-gram precision
ref_ngrams = Counter(
tuple(ref_tokens[i:i+n])
for i in range(len(ref_tokens) - n + 1)
)
cand_ngrams = Counter(
tuple(cand_tokens[i:i+n])
for i in range(len(cand_tokens) - n + 1)
)
overlap = sum((cand_ngrams & ref_ngrams).values())
total_cand_ngrams = max(sum(cand_ngrams.values()), 1)
precision = overlap / total_cand_ngrams
# Brevity penalty
ref_len = len(ref_tokens)
cand_len = len(cand_tokens)
brevity_penalty = min(1.0, ref_len / cand_len) if cand_len > 0 else 0.0
# Simplified BLEU (1-gram precision * brevity penalty)
return precision * brevity_penalty
if __name__ == "__main__":
reference_text = "the cat sat on the mat"
candidate_text = "the cat sat on mat"
print(f"BLEU score: {bleu_score(reference_text, candidate_text, n=2):.3f}")
Output
BLEU score: 0.500
How it works
This demo implements a stripped-down BLEU metric: it tokenizes both texts, builds n-gram counters, and computes precision as the overlap ratio. The brevity penalty adjusts for candidates shorter than the reference, discouraging overly terse outputs. Using Counter & Counter computes the minimum count per n-gram, exactly measuring matches. This simplified version ignores the geometric mean across multiple n-gram lengths and length-based smoothing, but it is enough to demonstrate the concept.
Common mistakes
- Using `json.loads` when you meant `json.load` for file objects
- Forgetting to lowercase or tokenize text consistently, skewing n-gram matches
- Dividing by zero when the candidate has no n-grams — guard with `max(..., 1)`
- Not applying a brevity penalty, so short guesses score too high
Variations
- Use `nltk.translate.bleu_score.sentence_bleu` for a full BLEU implementation
- Extend to multiple n-gram sizes (1..4) and average the log precision
Real-world use cases
- Quickly validating machine translation quality before deploying a model to production.
- A regression test for text generation, ensuring summaries or captions stay close to expected output.
- Comparing LLM responses against a reference answer to score RAG pipeline accuracy.
Sponsored
More from AI & LLM integration patterns
- Cache LLM Completions by Hashing the Prompt in Python easy
- Chain of Thought Prompting in Python: Step-by-Step Reasoning Demo easy
- Circuit Breaker Pattern in Python for LLM API Calls medium
- Cosine Similarity to Retrieve Top K Chunks in Python easy
- Demonstrate Prompt Injection Bypass in Python easy
- How to Accumulate Streamed Tokens into a Final String in Python easy
Keep learning
Related tutorials and quizzes for this topic.