Your chatbot ships, users complain, and the dashboards show nothing wrong. Free-form text has no single right answer to diff against, which is why teams use an LLM as a judge to put a number on quality. The LLM evaluation tutorial scored builds offline, before anyone shipped; this post drags that judge into production with agreement audits, canary sets, and CI gates.

The uncomfortable truth of shipping language models is that evaluation is the new system design. You do not architect your way to a reliable Large Language Model (LLM) feature, you measure your way there, one regression at a time. Think of a restaurant kitchen: offline evals are the head chef tasting a new dish in a quiet kitchen, which is not the same as the dish surviving a packed Friday night. Canary sets are the recipes you refuse to let slip, CI gates are the pass window where a failed plate never leaves, and online evals are you walking the floor watching which plates come back half-eaten.

“A model without a canary set is a model you will break on a Tuesday and only hear about from an angry customer on Thursday.”

Last Updated: July 2026 | Tested on: Python 3.14.6, deepeval 4.0.6, ragas 0.4.3, anthropic 0.111.0 | Difficulty: Advanced | Reading Time: 23 minutes

LLM as a judge is the practice of using one capable language model to grade the outputs of another against a written rubric. Instead of comparing to a fixed reference string, the judge reads the question, the answer, and any source context, then returns a score and a reason. It powers frameworks like DeepEval and Ragas, but the method is tool-independent: a rubric, a scale, and a way to check the grader against humans. That last part, checking the grader, is where most teams cut corners and get burned.

From Offline Scores to Production Evals

Offline evaluation is a single photograph: this build, this test set, this moment. Production evaluation is a loop that never stops turning, because a prompt tweak, a new model version from your provider, or a shift in what users ask can move quality without a single line of your code changing. The diagram below is the whole system on one page. Read it as a cycle: golden questions feed a judge, the judge feeds a gate that decides whether a build deploys, live traffic gets sampled after deploy, and the failures you find out there become tomorrow’s golden questions.

Two halves matter here. The offline half (left and top) is fast, deterministic, and runs on every pull request, so it has to be cheap and blunt. The online half is slow, noisy, and comes from real people, so it is the ground truth the offline half is only ever approximating. The arrow that closes the loop, mining down-voted prompts back into the canary set, is the one teams forget to draw, and it is the one that keeps your offline scores honest over time.

The Judge, Built as a Rubric

A judge is just a prompt with a job. You hand a strong model a rubric written in plain English, plus the thing to grade, and you ask for a score and a reason. There are two shapes worth knowing. Scoring grades one answer on an absolute scale, say 1 to 5 for faithfulness. Pairwise shows the judge two answers and asks which is better, which is what you want when comparing an old prompt against a new one. Pairwise is more reliable because “is A better than B” is an easier call than “is A a 3 or a 4”, but it costs you a comparison per pair instead of a single grade.

The code below is the real modern Anthropic SDK shape for both patterns. We did not execute it here because a live call needs an Application Programming Interface (API) key and spends money, but this is exactly what you would run, and the local stand-in scorers in the next sections let you reproduce the rest of the post with no key at all.

📄 judge.py: scoring and pairwise, one rubric each

import os, re
from anthropic import Anthropic

client = Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])
MODEL = "claude-opus-4-8"   # current at the time of writing; models change fast, check provider docs

SCORE_RUBRIC = """You are a strict grader. Score how FAITHFUL the answer is to the
context, 1 (invented facts) to 5 (fully supported). Reply as: score|one-line reason.
QUESTION: {q}
CONTEXT: {ctx}
ANSWER: {a}"""

PAIRWISE_RUBRIC = """Two answers to the same question. Reply with only 'A' or 'B',
whichever is more faithful to the context. If they tie, reply 'A'.
QUESTION: {q}
CONTEXT: {ctx}
ANSWER A: {a}
ANSWER B: {b}"""

def score_judge(q, ctx, a):
    msg = client.messages.create(model=MODEL, max_tokens=80, messages=[
        {"role": "user", "content": SCORE_RUBRIC.format(q=q, ctx=ctx, a=a)}])
    text = msg.content[0].text.strip()
    score = int(re.match(r"\s*(\d)", text).group(1))
    return score, text

def pairwise_judge(q, ctx, a, b):
    msg = client.messages.create(model=MODEL, max_tokens=5, messages=[
        {"role": "user", "content": PAIRWISE_RUBRIC.format(q=q, ctx=ctx, a=a, b=b)}])
    return msg.content[0].text.strip()[:1].upper()

s, reason = score_judge(
    q="Where was Python created?",
    ctx="Python was created by Guido van Rossum at CWI in the Netherlands.",
    a="Python was created by Guido van Rossum at Bell Labs.")
print(f"score={s}  reason={reason}")

▶ Example output (needs an API key, see note)

score=1  reason=1|The answer says Bell Labs, but the context states CWI in the Netherlands.

What happened here: The scoring judge caught the invented “Bell Labs” fact and returned a 1 with a reason you can act on. Notice the tie-break rule baked into the pairwise prompt: “if they tie, reply A”. That single line is a bias waiting to happen, because it means answer A wins every coin-flip. Which brings us to the question nobody asks often enough: is this judge actually any good?

Is Your Judge Any Good? Agreement and Bias

An LLM as a judge that you never audit is a rubber stamp. The honest test is simple: take a batch of outputs, have humans label them, have the judge label the same batch, and measure how often they agree. Raw agreement alone lies to you though, because if 90 percent of answers are fine, a judge that says “fine” every single time scores 90 percent while being useless. So we also compute Cohen’s kappa, which subtracts the agreement you would get by pure luck. The script below runs on 20 outputs with verdicts captured from a judge run, all stdlib, no key.

📄 judge_reliability.py: agreement and Cohen’s kappa

# 1 = faithful (grounded in the source), 0 = not faithful
judge_verdicts = [1,1,0,1,1,0,1,1,1,0,1,0,1,1,1,0,1,1,0,1]
human_labels   = [1,1,0,1,0,0,1,1,1,0,1,1,1,1,1,0,1,1,0,1]

n = len(judge_verdicts)
agree = sum(1 for j, h in zip(judge_verdicts, human_labels) if j == h)
agreement_rate = agree / n

# Cohen's kappa: (observed agreement - expected by chance) / (1 - expected)
p_obs = agreement_rate
pj1 = sum(judge_verdicts) / n
ph1 = sum(human_labels) / n
p_exp = pj1 * ph1 + (1 - pj1) * (1 - ph1)
kappa = (p_obs - p_exp) / (1 - p_exp)

print(f"outputs graded      : {n}")
print(f"judge == human      : {agree}")
print(f"raw agreement rate  : {agreement_rate:.0%}")
print(f"expected-by-chance  : {p_exp:.0%}")
print(f"Cohen's kappa       : {kappa:.2f}")
verdict = "trust the judge" if kappa >= 0.6 else "judge needs a tighter rubric"
print(f"decision            : {verdict}")

▶ Output

outputs graded      : 20
judge == human      : 18
raw agreement rate  : 90%
expected-by-chance  : 58%
Cohen's kappa       : 0.76
decision            : trust the judge

What happened here: Raw agreement is a comfortable 90 percent, but chance alone would have given you 58 percent, so the judge is really only earning the 32 points above luck. Kappa boils that down to 0.76, which lands in the “substantial agreement” band and clears our 0.6 bar. If kappa had come back at 0.3, that 90 percent would have been a mirage from an imbalanced dataset, and you would tighten the rubric before trusting a single score. Published work like MT-Bench found strong judges agree with humans around 80 percent of the time, so treat anything wildly higher on your own data with suspicion.

Agreement is only half the audit. LLM judges carry three biases you can actually measure. The most dangerous is position bias, where the judge leans toward whichever answer it sees first, exactly the tie-break trap from our pairwise prompt. The fix is a harness that runs each comparison in both orders and counts how often the verdict flips.

📄 bias_traps.py: measuring position bias by swapping order

def mock_judge(first, second):
    # a well-behaved judge picks on quality alone; this stand-in leans
    # slightly toward the FIRST slot when the two answers are close.
    return "A" if first["quality"] + 0.03 >= second["quality"] else "B"

# 10 pairs: a few near-ties (where the thumb decides), most are clear wins.
pairs = [
    ({"quality": 0.70}, {"quality": 0.71}), ({"quality": 0.80}, {"quality": 0.70}),
    ({"quality": 0.66}, {"quality": 0.68}), ({"quality": 0.55}, {"quality": 0.63}),
    ({"quality": 0.90}, {"quality": 0.78}), ({"quality": 0.60}, {"quality": 0.62}),
    ({"quality": 0.75}, {"quality": 0.66}), ({"quality": 0.85}, {"quality": 0.70}),
    ({"quality": 0.58}, {"quality": 0.65}), ({"quality": 0.82}, {"quality": 0.71}),
]

flips = 0
for a, b in pairs:
    original = mock_judge(a, b)                 # A shown first
    swapped = mock_judge(b, a)                  # same pair, order flipped
    swapped_in_original_terms = "A" if swapped == "B" else "B"
    if original != swapped_in_original_terms:
        flips += 1

print(f"pairs compared         : {len(pairs)}")
print(f"verdict flipped on swap: {flips}")
print(f"position-bias rate     : {flips / len(pairs):.0%}  (0% = order-independent)")

▶ Output

pairs compared         : 10
verdict flipped on swap: 3
position-bias rate     : 30%

What happened here: On the three near-tie pairs the verdict flipped when we swapped the order, so 30 percent of this judge’s calls depended on position, not quality, and a real judge shows the same pattern on close cases. The fix is cheap: run every comparison in both orders and only count a win when both agree. Here are the three biases worth building traps for.

Canary Sets: Questions That Must Never Regress

Miners used to carry a canary into the tunnel. The bird was more sensitive to bad air than a human, so if it stopped singing, you got out before you knew anything was wrong. A canary set is that bird for your model: a small, curated list of questions where you know the correct answer cold, and where a wrong answer would be embarrassing or expensive. When one of them regresses, the build stops, no debate.

The best canaries are harvested, not invented. Every time a user reports a wrong answer or you catch a hallucination in the logs, that becomes a canary with its required facts pinned down. Thirty solid canaries mined from real incidents beat three hundred synthetic ones. Each canary carries the exact facts a correct answer must contain, so the cheap tier can check grounding with no LLM in the loop.

📄 canaries.py: golden questions with the facts they must contain

# Mined from real support tickets where the bot got it wrong at least once.
# must_have lists the facts a correct answer cannot drop.
CANARIES = [
    {"q": "What header carries the API key?",       "must_have": ["x-api-key"]},
    {"q": "Which city hosts the primary database?", "must_have": ["frankfurt"]},
    {"q": "What is the free-tier request cap per day?", "must_have": ["1000", "day"]},
    {"q": "How do refunds get issued?",             "must_have": ["original", "payment"]},
]

What happened here: This is data, not a program run, so there is no output to show. Each entry pins the non-negotiable facts for one question. The keyword check is the cheap, deterministic tier you can run on every commit with zero API cost. For nuance a keyword match cannot catch, like tone or partial correctness, you layer an LLM judge on top for the nightly run. Cheap-and-blunt on every push, slow-and-smart on a schedule.

Evals as CI Gates

A gate is the pass window in the kitchen: a plate either clears inspection or it goes back. In software terms, the eval suite runs, and if the canary pass rate drops below your threshold, the build fails with a non-zero exit code and the deploy is blocked. The script below is the real gate logic, running the canary set against two builds of a bot. Build A is healthy. Build B shipped after a careless prompt edit that dropped the exact numbers and the refund rule, the kind of change that sails through code review because the code did not change.

📄 canary_gate.py: fail the build when a canary regresses

import sys
from canaries import CANARIES

good_bot = {   # build A: answers with the facts intact
    "What header carries the API key?":
        "Send it in the x-api-key header on every request.",
    "Which city hosts the primary database?":
        "The primary database runs in Frankfurt.",
    "What is the free-tier request cap per day?":
        "The free tier allows 1000 requests per day.",
    "How do refunds get issued?":
        "Refunds go back to the original payment method within 5 days.",
}
degraded_bot = {   # build B: the prompt edit made it vague, numbers dropped
    "What header carries the API key?":
        "Send it in the x-api-key header on every request.",
    "Which city hosts the primary database?":
        "It is hosted in one of our EU regions.",
    "What is the free-tier request cap per day?":
        "The free tier includes a generous daily allowance.",
    "How do refunds get issued?":
        "Our support team handles refunds on request.",
}

def score(bot, threshold=0.9):
    passed = 0
    for c in CANARIES:
        answer = bot[c["q"]].lower()
        ok = all(fact in answer for fact in c["must_have"])
        passed += ok
        print(f"  [{'PASS' if ok else 'FAIL'}] {c['q']}")
    rate = passed / len(CANARIES)
    gate = rate >= threshold
    print(f"  canary pass rate: {rate:.0%}  (gate needs >= {threshold:.0%}) -> "
          f"{'GATE OPEN' if gate else 'GATE BLOCKED'}")
    return gate

print("== build A: current bot ==")
a_ok = score(good_bot)
print("== build B: after prompt edit ==")
b_ok = score(degraded_bot)
sys.exit(0 if (a_ok and b_ok) else 1)   # non-zero exit fails the CI job

▶ Output (process exit code was 1)

== build A: current bot ==
  [PASS] What header carries the API key?
  [PASS] Which city hosts the primary database?
  [PASS] What is the free-tier request cap per day?
  [PASS] How do refunds get issued?
  canary pass rate: 100%  (gate needs >= 90%) -> GATE OPEN
== build B: after prompt edit ==
  [PASS] What header carries the API key?
  [FAIL] Which city hosts the primary database?
  [FAIL] What is the free-tier request cap per day?
  [FAIL] How do refunds get issued?
  canary pass rate: 25%  (gate needs >= 90%) -> GATE BLOCKED

What happened here: Build A passed all four canaries and the gate opened. Build B answered vaguely (“one of our EU regions” instead of “Frankfurt”, “a generous daily allowance” instead of “1000 per day”), so three canaries failed, the pass rate cratered to 25 percent, and the script exited with code 1. In CI a non-zero exit is a red build and the deploy never happens, so the regression was caught by a machine at 2am, not by a customer.

In a real repo you would wire this through pytest so the reporting and thresholds are standard. DeepEval ships a pytest plugin built exactly for this, and the GitHub Actions workflow that runs it is a few lines. The Python code runs the LLM-judge tier of your canary suite, the YAML runs it on every pull request.

📄 test_canaries.py + ci.yml: the pytest gate in GitHub Actions

# test_canaries.py  (run with: deepeval test run test_canaries.py)
import pytest
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import FaithfulnessMetric

CASES = [
    ("Which city hosts the primary database?",
     "The primary database runs in Frankfurt.",
     ["The primary production database is hosted in Frankfurt."]),
]

@pytest.mark.parametrize("q,answer,ctx", CASES)
def test_canary_faithfulness(q, answer, ctx):
    case = LLMTestCase(input=q, actual_output=answer, retrieval_context=ctx)
    assert_test(case, [FaithfulnessMetric(threshold=0.8)])  # below 0.8 -> build fails

📄 .github/workflows/ci.yml

# .github/workflows/ci.yml
name: eval-gate
on: [pull_request]
jobs:
  canary:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.14" }
      - run: pip install deepeval pytest
      - run: deepeval test run test_canaries.py    # non-zero exit blocks the merge
        env: { ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} }

What happened here: The pytest file turns each canary into a parametrized test with a faithfulness threshold, so a failing eval reads like a failing assertion. The workflow installs DeepEval and runs it on every pull request. A green check means the canaries held and the merge is allowed; a red X means a canary regressed and the merge button stays disabled. The key from repository secrets keeps it out of the code, which matters because the judge tier does spend tokens on every run.

Online Evals and the Offline Gap

Here is the trap that catches confident teams. Your offline suite is green, every canary passes, so you relax. Meanwhile real users are quietly unhappy, because the questions they actually ask look nothing like your tidy test set. Online evals close that blind spot: you sample a slice of live outputs, collect a signal (the humble thumbs up or down works remarkably well), and optionally run an LLM as a judge over the sample. Then you compare that against your offline scores. When the two drift apart, your test set has gone stale.

📄 drift.py: offline judge score vs online thumbs signal

# offline_scores: judge faithfulness on the canary set, normalized 0..1.
# thumbs: the up/down signal from real users on live answers.
offline_scores = [0.92, 0.88, 0.95, 0.90, 0.87, 0.93, 0.89, 0.91]
thumbs = ["up","up","down","up","down","down","up","down","down","up","down","down"]

offline_mean = sum(offline_scores) / len(offline_scores)
online_rate = thumbs.count("up") / len(thumbs)
gap = offline_mean - online_rate

print(f"offline judge score (mean): {offline_mean:.0%}")
print(f"online thumbs-up rate     : {online_rate:.0%}")
print(f"offline / online gap      : {gap:+.0%}")
if gap > 0.15:
    print("signal: your canary set is not representative of real questions")
    print("action: mine the down-voted prompts and add them as new canaries")

▶ Output

offline judge score (mean): 91%
online thumbs-up rate     : 42%
offline / online gap      : +49%
signal: your canary set is not representative of real questions
action: mine the down-voted prompts and add them as new canaries

What happened here: Offline said 91 percent, users said 42 percent, a 49-point gap that means your test set and your traffic have drifted apart. The offline number is not wrong, it is just answering a question nobody is asking. The action closes the loop from the diagram: pull the down-voted prompts, turn them into fresh canaries, and the offline suite starts tracking reality again. A growing gap is your earliest warning that the world moved and your evals did not.

Retrieval Metrics Primer: Recall@K, MRR, nDCG

When your app is retrieval-augmented, half of quality is decided before the model writes a word: did retrieval put the right chunks in front of it? Three metrics answer that, and they are the bridge to the retrieval and agentic Retrieval-Augmented Generation (RAG) posts coming next. Picture a librarian handing you a stack of books for a question. Recall@K asks how many of the truly useful books made it into the top K of the stack. MRR (Mean Reciprocal Rank) asks how near the top the first useful book sat. nDCG rewards useful books and rewards them extra for being ranked high, because a great chunk buried at position 9 barely helps.

📄 retrieval_metrics.py: Recall@K, MRR, and nDCG from scratch

import math

# 1 = the chunk at this rank is relevant, 0 = not. One query's ranked result.
ranked = [0, 1, 0, 1, 1]     # positions 1..5, best-guess first
total_relevant = 4           # 4 relevant chunks exist in the whole corpus
K = 5

def recall_at_k(ranked, total_relevant, k):
    return sum(ranked[:k]) / total_relevant

def mrr(ranked):
    for i, rel in enumerate(ranked, start=1):
        if rel:
            return 1 / i
    return 0.0

def ndcg_at_k(ranked, k):
    dcg = sum(rel / math.log2(i + 1) for i, rel in enumerate(ranked[:k], start=1))
    ideal = sorted(ranked, reverse=True)[:k]
    idcg = sum(rel / math.log2(i + 1) for i, rel in enumerate(ideal, start=1))
    return dcg / idcg if idcg else 0.0

print(f"ranked relevance : {ranked}")
print(f"Recall@{K}        : {recall_at_k(ranked, total_relevant, K):.2f}")
print(f"MRR              : {mrr(ranked):.2f}")
print(f"nDCG@{K}          : {ndcg_at_k(ranked, K):.2f}")

▶ Output

ranked relevance : [0, 1, 0, 1, 1]
Recall@5        : 0.75
MRR              : 0.50
nDCG@5          : 0.68

What happened here: Three of the four relevant chunks landed in the top five, so Recall@5 is 0.75. The first relevant chunk sat at position 2, so MRR is 1/2 = 0.50, meaning the top slot was wasted on an irrelevant chunk. nDCG@5 of 0.68 reflects that the relevant chunks were present but not ranked as high as the ideal ordering would put them. Read together: retrieval is finding most of the right material but ranking it poorly, which points you at your reranker, not your embedding model. These three numbers are how you will grade the RAG systems in the posts ahead.

Common Mistakes

Conclusion

You started with an offline score and ended with a living eval system built around an LLM as a judge. You built a scoring and a pairwise judge, audited the judge itself with agreement and Cohen’s kappa, measured position bias by swapping order, and turned canary sets mined from real failures into a CI gate that blocked a degraded build with a red exit code. Then you looked past offline entirely, caught a 49-point offline/online gap, and learned the three retrieval metrics that grade the RAG systems coming next. The theme underneath all of it: evaluation is the new system design, and the team that measures fastest ships the most reliable model.

Next up is LLM observability with Langfuse, where the traces and spans give you the raw material these online evals feed on. Want the bigger picture? Browse the complete Python + AI/ML tutorial series home to see where LLM-as-a-judge fits among every other lesson, from your first script to production AI systems.

Frequently Asked Questions

Interview Questions on LLM Evaluation

Interviewers rarely ask for definitions. They ask what happens in situations like these.

Q: Why measure Cohen’s kappa instead of just raw agreement between a judge and human labels?

Raw agreement is inflated whenever the labels are imbalanced. If 90 percent of answers are fine, a judge that blindly says “fine” scores 90 percent agreement while carrying zero real skill. Kappa subtracts the agreement you would expect from chance given each side’s base rates, so it exposes that lazy judge as near-zero. A high agreement paired with a low kappa is the classic tell of an unbalanced dataset, and it is exactly the case where you should not trust the judge yet.

Q: What is position bias in a pairwise LLM judge, and how do you detect and fix it?

Position bias is the judge leaning toward whichever answer is presented first (or last) rather than deciding on quality. You detect it by running every comparison twice with the answer order swapped and counting how often the verdict flips; a nonzero flip rate on near-tie pairs is position bias. You fix it by requiring both orderings to agree before counting a win, or by switching to absolute scoring where each answer is graded on its own so order never enters the decision.

Q: What makes a good canary set, and where should the questions come from?

A good canary set is small, high-signal, and mined from real failures rather than invented. Each canary pins the exact facts a correct answer must contain, so a regression is unambiguous. The questions should come from actual incidents: reported wrong answers, hallucinations caught in logs, and down-voted responses. Thirty canaries harvested from real breakage catch more regressions than hundreds of synthetic ones, because they carry the phrasings and edge cases users actually produce.

Q: How would you turn an eval suite into a CI gate that blocks a bad deploy?

Express canaries as tests with a pass threshold, run them in the pipeline, and make a failing eval exit non-zero so the CI job goes red and the merge or deploy is blocked. A tool like DeepEval’s pytest plugin gives you standard reporting for free. Split it into tiers: a cheap deterministic check on every push and a slower LLM-judge run on a schedule, so token cost stays bounded while still catching the regressions that only a judge can see.

Q: Your offline evals are all green but user satisfaction is falling. What is happening and what do you do?

That is an offline/online drift: the offline suite is answering questions real users are not asking, so its high scores are honest but irrelevant. The fix is to close the loop. Sample live traffic, collect a signal like thumbs up or down, compute the gap between offline scores and online satisfaction, then mine the down-voted prompts into new canaries. Once real failures are represented in the offline set, the two numbers converge again and your gate regains its meaning.

Q: Explain Recall@K, MRR, and nDCG, and what each one tells you about a retriever.

Recall@K is the fraction of all relevant chunks that appear in the top K, so it measures coverage: did retrieval find the material at all? MRR is the reciprocal rank of the first relevant chunk, so it measures how quickly the user hits something useful. nDCG rewards relevant chunks and rewards them more for ranking high, so it measures ordering quality. High recall with low MRR or nDCG means retrieval is finding the right chunks but ranking them poorly, which points at the reranker rather than the embedding model.

Series: Python + AI/ML Cookbook, Part 6: GenAI & LLMs

Go deeper: when you outgrow this post, Hugging Face documentation is the next stop.

Related Posts

Previous: How to Evaluate LLM Output: RAGAS, DeepEval, Hallucination Detection

Next: LLM Observability with Langfuse: Traces, Spans, and Cost

Series Home: Python + AI/ML Tutorial Series