How to Create a Mock LLM Judge Rubric Score in Python

Scores a response against a rubric by counting keyword matches, returning total, percentage, and per-criterion feedback.

Easy Python 3.9+ Aug 9, 2026 AI & LLM integration patterns 15 views 0 copies

llm evaluation rubric mock scoring

Python code

40 lines

Python 3.9+

def judge_score(response, rubric):
    """Mock LLM judge that scores a response against a rubric."""
    total = 0
    max_total = 0
    feedback = []

    for criterion, rubric_item in rubric.items():
        max_points = rubric_item["max"]
        description = rubric_item["description"]

        # Simple mock scoring: score based on keyword presence
        keywords = rubric_item.get("keywords", [])
        matches = sum(1 for kw in keywords if kw.lower() in response.lower())
        score = min(matches, max_points)

        total += score
        max_total += max_points
        feedback.append(f"{criterion}: {score}/{max_points} ({description})")

    return {
        "total_score": total,
        "max_score": max_total,
        "percentage": round(total / max_total * 100, 1),
        "feedback": feedback
    }


if __name__ == "__main__":
    rubric = {
        "relevance": {"max": 3, "description": "Response addresses the topic", "keywords": ["python", "code"]},
        "clarity": {"max": 2, "description": "Response is clear and concise", "keywords": ["simple", "easy"]},
        "completeness": {"max": 2, "description": "Response covers key points", "keywords": ["function", "main"]}
    }

    response = "Python functions are simple and easy to use, this code shows main structure."

    result = judge_score(response, rubric)
    print(f"Total: {result['total_score']}/{result['max_score']} ({result['percentage']}%)")
    for line in result["feedback"]:
        print(f"  - {line}")

Output

stdout

Total: 6/7 (85.7%)
  - relevance: 3/3 (Response addresses the topic)
  - clarity: 2/2 (Response is clear and concise)
  - completeness: 2/2 (Response covers key points)

How it works

This function takes a response and a rubric dictionary. For each criterion, it counts how many keywords appear (case-insensitively) in the response, then caps the score at the maximum points. The total score and percentage are computed, and feedback is built for each criterion. The mock returns a structured dict that mimics an LLM judge's output, making it easy to swap in a real model later. This pattern is useful for prototyping evaluation pipelines without calling external APIs.

Common mistakes

Forgetting case-insensitivity when checking keywords, causing missed matches.
Not capping the score at the max, allowing over-scoring.
Assuming keywords are present in the rubric, leading to KeyError.
Using string concatenation for feedback instead of f-strings.

Variations

Use `sum(1 for kw in keywords if kw in response)` for a simpler but case-sensitive check.
Replace keyword matching with a call to an actual LLM API to return scores.

Real-world use cases

Prototyping an automated essay grader that checks for key terms before calling a paid LLM.
Building a unit-test harness for prompt templates that verifies expected keywords appear in generated responses.
Creating a lightweight quality gate in CI that flags pull request descriptions missing important topics.

How to Create a Mock LLM Judge Rubric Score in Python

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from AI & LLM integration patterns

Tutorials

Quizzes

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from AI & LLM integration patterns

Keep learning

Tutorials

Quizzes