evals

An eval is a versioned set of cases, not a table of scores. Seeing the shape of a case answers the question everyone postpones: what exactly is being scored, and who decides that it is good.

1. 01

   The shape of a case

   A case carries its input, its judges and its origin. All three matter, and the third more than one might think: a case whose reason for existing has been forgotten is a case nobody dares modify or delete.

   jsCopy

   {
     id: 'cap-2024',
     input: 'What is my reimbursement cap?',
     origin: 'incident of 2026-03-14',    // every fixed defect leaves a case
   
     // “right answer” does not exist: three criteria, scored separately,
     // because they do not improve together
     judges: [
       exact((r) => r.amount === 2400),                  // programmed: safe, narrow
       rule((r) => r.cites('contract-2024.pdf')),        // programmed: verifiable
       model('Does the answer invent a figure?'),        // model: broad, biased
     ],
   }
   

   exact

   The cheapest and most reliable judge, when it applies. The whole skill consists in phrasing enough cases for it to apply often.

   model

   The model judge: indispensable for what cannot be compared character by character, and the only one of the three whose mark is worth nothing until it has been set against a sample corrected by hand. Its biases are documented; the `juge-automatique` entry details them.

   The trapThe “origin” field is what makes it possible to renew the set from reality instead of letting it grow old. Without it, nobody knows which incident a line was protecting, so nobody dares remove or replace it, and the set freezes.

2. 02

   Scoring a path, not an arrival

   Evaluating an agent calls for replays: two runs of the same case diverge. You therefore read rates and not verdicts, and you look at the journey as much as at the answer, because a goal that has been reached may have been reached in an unacceptable way.

   jsCopy

   const runs = await Promise.all(
     Array.from({ length: 10 }, () => execute(testCase)),   // the same case, ten times
   );
   
   const report = {
     success: rate(runs, (r) => r.ok),                  // a rate, not a verdict
     turns: median(runs, (r) => r.turns),               // what the path costs
     wrong_tool_calls: rate(runs, (r) => r.calls.some((c) => !c.useful)),
     irreversible_without_approval: runs.filter((r) => r.actedWithoutApproval).length,
   };
   
   // an agent that reaches the goal in twenty turns and deletes a file along
   // the way has not succeeded: the success rate alone would never say so.
   if (report.irreversible_without_approval > 0) fail(report);
   

   length: 10

   The number of replays. It does not have to be large, it has to be constant: what informs you is the comparison from one version to the next, not the absolute value.

   irreversible_without_approval

   The only criterion of the batch that is a count rather than a rate, because it does not average out: a single occurrence is enough to fail the version.

   The trapThis count fails a version on its own, and that is deliberate: on actions with no way back, a rate means nothing, since it would stay excellent while letting through the very case you were trying to prevent.

What variesEvaluation tooling each exposes its own vocabulary, and the names used here are those of common sense, not of a product. What does not vary: several criteria scored separately rather than a single score, a replay so as to read rates as soon as an agent is involved, a model judge that itself has to be checked, and cases that come from real incidents.