How to compute exact match metric in Python

Computes the exact match (EM) metric for LLM outputs by normalizing text and comparing predictions against references.

Easy Python 3.9+ Aug 9, 2026 AI & LLM integration patterns 12 views 0 copies

exact-match metric evaluation nlp llm

Python code

17 lines
Python 3.9+
Run it 
def compute_exact_match(predictions, references):
    def normalize(text):
        import re
        text = text.lower().strip()
        text = re.sub(r'\b(a|an|the)\b', ' ', text)
        text = re.sub(r'[^a-z0-9\s]', '', text)
        text = ' '.join(text.split())
        return text

    matches = sum(1 for pred, ref in zip(predictions, references) if normalize(pred) == normalize(ref))
    return matches / len(predictions) if predictions else 0.0

if __name__ == "__main__":
    predictions = ["The quick brown fox", "jumps over the lazy dog", "Hello World!"]
    references = ["quick brown fox", "jumps over lazy dog", "hello world"]
    em = compute_exact_match(predictions, references)
    print(f"Exact Match: {em:.2f}")

Output

stdout 
Exact Match: 1.00

How it works

The exact match metric is a strict evaluation used in NLP tasks like question answering. The normalize function lowercases text, strips common articles, removes punctuation, and collapses whitespace so minor formatting differences don't cause false mismatches. The function computes the ratio of predictions that exactly match their reference after normalization. It falls back to 0.0 when the predictions list is empty, avoiding a division by zero. This gives a single float between 0 and 1 that represents how often the model completed the task perfectly.

Common mistakes

  • Not handling empty predictions list, causing ZeroDivisionError
  • Forgetting to normalize both predictions and references consistently
  • Over-normalizing by removing words that carry semantic meaning
  • Using zip without ensuring predictions and references have equal lengths

Variations

  1. Use `sklearn.metrics.accuracy_score` after converting to binary matches
  2. Apply different normalization rules like stemming or lemmatization for domain-specific text

Real-world use cases

  • Evaluating question-answering model outputs against gold-standard answers on SQuAD-style benchmarks.
  • Measuring how often a text-generation system produces byte-perfect responses for form-filling or data extraction tasks.
  • Scoring code completion or translation models where exact output fidelity matters before accepting into production.

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from AI & LLM integration patterns

Keep learning

Related tutorials and quizzes for this topic.

Tutorials

Quizzes