GlossaryFloor 3 · The Agentthe block caught in a loop: it starts again until it gets thereFloor 3 · The Agent
LLM as a judge
No. 105 · v2026-08FR: juge automatiqueAn LLM as a judge is a model tasked with scoring another system’s answers against written criteria, where no mechanical comparison can do the job. Like a marker to whom you hand an essay along with the marking scheme: they are quick and they never tire, but the mark is only worth what the scheme is worth.
What it is not
An LLM as a judge does not replace human review, it displaces it. What it does is apply at scale a judgement that a person had to formulate first, and then calibrate against a sample marked by hand: without that sample you measure nothing, you merely collect marks. Nor is it a benchmark, which compares models on an examination common to the whole trade, whereas a judge scores your answers against your criteria. And it is not a neutral arbiter, since it carries the same blind spots as what it evaluates as soon as the same model holds both roles.
In depth
The mechanism
The arrangement is one call to a language model whose input is an answer to be scored and whose output is a verdict. It exists because most of the criteria that matter cannot be checked by string comparison: an open answer can be right in twenty different wordings, and wrong in a wording very close to the right one. Its most solid form is not an overall mark but a series of criteria scored separately, each binary or on a short scale, with a quotation of the passage that justifies every verdict. Requiring that quotation changes the nature of the exercise: it makes the verdict checkable by sampling, and it forbids the judge from scoring an impression. Two arrangements coexist: absolute scoring, which judges one answer on its own against criteria, and pairwise comparison, which names the better of two answers. The second is the more stable, because it is easier to say which of the two is better than to place an answer on a scale; the first has the advantage of comparing one version with the next.
The two gestures
What an LLM as a judge changes is volume and delay: it makes it possible to replay a whole set of cases at every modification, where human review is paid for in hours and arrives only once the system is in service. That is what makes it the third piece of an eval, alongside the set of cases and the criterion, as soon as the output being measured is free text. Making it serious rests on two gestures, and the first is the one most often left out: the criteria are written before seeing the answers, failing which you draft the scheme that validates what you have already produced. The second is calibration: you have a person mark a sample of a few dozen cases, you compare with the judge, and you grant it trust only up to the level of agreement observed. That agreement is also measured between two people, which gives the real ceiling of the exercise: a judge that matches the agreement rate of two human reviewers has done everything that could be asked of it. What remains is to decide the fate of the cases where it hesitates, and the cheapest answer is to route them to a person rather than to force a verdict.
The known biases
The biases of an LLM as a judge are documented and recur from one arrangement to the next. It scores generously, with a distribution bunched towards the top of the scale, which makes two versions indistinguishable. It prefers long, detailed, well-presented answers, including where the length adds falsehood. Order of presentation weighs on a pairwise comparison, one position receiving an advantage that owes nothing to the content, which makes it necessary to replay every pair both ways and to keep only the stable verdicts. It also lets itself be swayed by its own manner of writing, and prefers answers that resemble what it would have produced. Finally, it shares the blind spots of what it evaluates when the same model sits on both sides: an error it makes itself is an error it does not see, which is precisely the case where you would need it most. The prudent rule therefore fits into one sentence: an uncalibrated judge measures nothing, and a table of scores produced by an uncalibrated judge is more dangerous than no measurement at all, because it gives the feeling of having measured.
Under the hood2 steps · the real shape of the objects
What separates a serious judge from an opinion at scale is not its score, it is the contract you require it to fill. The grid goes in on one side, a justified verdict comes out of the other, and both shapes are what this entry claims.
- 01
The grid, written first
Each criterion is a closed question, checkable by searching the text. That is what stops the judge describing what is in front of it instead of measuring it, and it is why the grid is drafted before the work has been read.
[ { "id": "sources", "question": "Is every figure given accompanied by its source?" }, { "id": "reco", "question": "Does the recommendation fit in one findable sentence?" }, { "id": "objections", "question": "Are the predictable objections addressed?" } ]- question
- Closed, always. “Is it clear?” cannot be checked and produces a mark that reflects the judge; “does the recommendation fit in one findable sentence?” is checked by looking for it.
- id
- It makes marks comparable from one piece of work to the next and from one version to the next. A grid whose criteria change along the way no longer measures anything.
The trapThe order matters as much as the content. A grid written after reading conforms to the text read: it describes a taste, and it will give everyone the same marks.
- 02
The verdict, and its evidence
The judge does not return a number, it returns a position per criterion with the exact quotation that justifies it. That quotation is what makes the judgement arguable: you do not argue with the mark, you argue with the evidence.
[ { "criterion": "sources", "verdict": "no", "quotation": "the market grew by half last year" }, { "criterion": "reco", "verdict": "yes", "quotation": "We recommend keeping the supplier until June." } ]- verdict
- Three values at most, never a fine scale: a judge marking out of ten invents a precision it does not have, and the average drawn from it erases the criterion you cared about.
- quotation
- Taken word for word from the work being marked, which makes it checkable in three seconds. A quotation that cannot be found in the text is the most reliable alarm signal available.
- a missing quotation
- It must count as “no”, never as “partly”. That is the rule that blocks the generous mark, because it turns absent evidence into an unfavourable position.
The trapNone of this says the judge is right. The contract makes its error visible and arguable, it does not prevent it: which is why calibrating against a sample marked by a person remains the precondition.
- evalsthe shape of a case, and what marking a path rather than an arrival means
- structured outputwhat a schema really promises, and why hoping for a shape is not imposing one
What variesThe field names here are inventions for demonstration, and every team adopts its own. What does not vary: without an exact quotation a mark is not auditable, a fine scale manufactures a precision that does not exist, and a judge sharing the model of what it evaluates shares its blind spots too.
Relations where the neighbours live
Check 3 questions · click your answer
Level 1 · Recognise
Why entrust the scoring of answers to a model rather than to an automatic comparison?
Level 2 · Distinguish
Your automatic judge has never been compared with human marking. What are its scores worth?
Level 2 · Distinguish
Why avoid having a system scored by the very model that makes it run?
Who works with this 1 role
The roles for which this term is part of the ordinary work.
Lexigraph, "LLM as a judge", v2026-08, https://www.lexigraph.org/en/llm-as-a-judge/, CC BY 4.0.