structured output

A structured output is a schema plus a constraint. The schema can be read; the constraint plays out at one precise point in the production of the text, and it is that point which separates a hoped for shape from a guaranteed one.

1. 01

   The schema, and what it promises

   The schema describes the expected shape, field by field. It travels with the request, which makes it a cost per turn rather than a setting made once: lengthening a schema is paid for on every call.

   jsonCopy

   {
     "name": "invoice_verdict",
     "schema": {
       "type": "object",
       "properties": {
         "amount_excl_tax": { "type": "number" },
         "currency": { "type": "string", "enum": ["EUR", "USD", "CHF"] },
         "issue_date": { "type": "string", "format": "date" },
         "confidence": { "type": "string", "enum": ["certain", "unsure", "unreadable"] }
       },
       "required": ["amount_excl_tax", "currency", "issue_date", "confidence"],
       "additionalProperties": false
     },
     "strict": true
   }
   

   enum

   A closed list. It is the most useful constraint of the lot: it makes a value outside the list impossible, where an instruction in plain language would only make it improbable.

   confidence

   The way out, and the field everyone forgets. Uncertainty is planned for in the schema, failing which it comes back out disguised as a value, and gets discovered much later in the data.

   strict

   The flag that switches from a requested shape to an imposed one. Everything turns on it, and the next step shows why.

   The trapThis is the rarely measured cost layer 2 mentions, and it can be seen here: the constraint applies from the very first fragment produced. Hence the third step, which separates the two moments.

2. 02

   Hoping, or imposing

   Here is the difference, at the exact point where it occurs. At each step the model proposes a distribution over the possible tokens; constraining consists in cancelling, before the draw, those that would lead outside the schema. So it is not a check after the fact: it is an amputation of the outcomes.

   jsCopy

   // A · the shape REQUESTED: you beg, then you check, then you start over
   let raw = await model.answer({ messages: [...msgs, instruction('Reply in JSON.')] });
   let parsed = tryToRead(raw);              // may fail: text before, trailing comma...
   if (!parsed) raw = await model.answer({ messages: [...msgs, complaint(raw)] });
   
   // B · the shape IMPOSED: at each step, tokens outside the schema are discarded
   function nextToken(scores, schemaState) {
     const allowed = automaton.acceptableTokens(schemaState);  // derived from the schema
     for (const t of Object.keys(scores)) {
       if (!allowed.has(t)) scores[t] = -Infinity;             // <- the outcome vanishes
     }
     return draw(scores);
   }
   

   tryToRead

   All the defensive code that a real constraint makes pointless: a preamble to strip, a code block to unwrap, one comma too many, fanciful quotation marks.

   -Infinity

   The token becomes impossible, not improbable. It is the only guarantee of shape that exists, and it is not obtained by asking more politely.

   The trapIt is the same mechanics as a tool call, down to the destination, as layer 2 says. A provider that accepts a schema for its tools but not for its responses therefore does not have a technical limit, it has a limit in the surface it exposes.

3. 03

   What the shape does not say

   A conforming response remains a plausible production. This is the central trap of the entry, and here it is in its most concrete form: the check on the shape has passed, and nothing has been verified yet.

   jsCopy

   const r = { amount_excl_tax: 1240.00, currency: 'EUR',
               issue_date: '2027-13-45', confidence: 'certain' };
   
   validate(r, schema).ok;   // true  <- the SHAPE is beyond reproach
   
   // what is left to do, and what nobody does in place of the business
   const checks = [
     () => isARealDate(r.issue_date),                       // false: month 13, day 45
     () => r.amount_excl_tax > 0 && r.amount_excl_tax < 1e6, // ceiling of the domain
     () => existsInRecords(r.supplier_number),              // does the identifier exist?
     () => r.confidence !== 'certain' || sourceIsLegible(),  // confidence is declared,
   ];                                                        // so it proves nothing
   
   if (!checks.every((c) => c())) reject(r);
   

   The trapThe fourth line is the most treacherous: a confidence field is filled in by the same mechanism as the rest, so “certain” is not a measurement, it is one more prediction. It serves to collect a doubt when there is one, never to do without one.

What variesField names vary: the schema is declared sometimes under “response_format”, sometimes under a tool name, and the flag for a real constraint changes name from one provider to another. What does not vary: the constraint, when it is real, is enforced as each fragment is drawn and not after, and no schema, however strict, passes the slightest judgement on the truth of the values it frames.