Interactive

Audit an LLM judge

Drag the sliders to give the judge the biases real LLM evaluators are known to have, and watch standard measurement checks catch them. Everything recomputes live, on 300 simulated items, in your browser.

The judge's behavior

Verbosity bias

How much a longer answer raises the score, regardless of quality.

Position pull

In head-to-head mode, how strongly the judge favors whichever answer it sees first.

Inconsistency

Run-to-run randomness. Higher means the judge disagrees with itself on a re-score.

Group tilt

A systematic offset applied to group B's answers at equal quality.

The audit

Self-consistencytest-retest, weighted κ · humans ≈ 0.90

Agreement with humansweighted κ vs consensus

Verbosity biasscore ~ length | quality · humans ≈ +0.13

Group invariancescore gap at equal quality

Position biasfirst-shown answer wins · 50% is fair

How to read this. The "true quality" of each answer is known, which is what lets the audit tell real signal from bias. A judge can agree with humans most of the time and still fail these checks: rewarding length, drifting between groups, or flipping its verdict when you swap the order of two answers.

The judge here is simulated to reproduce documented behaviors, so it's reproducible with no API. The same audit runs unchanged on a live model. Full methodology and code: auditing-the-llm-judge. The argument behind it: AI Evaluation Is a Measurement Problem.