determinism

In depth

Two notions to separate

Two notions often overlap and are worth separating: determinism, which holds that the same input gives the same output here and now, and reproducibility, which holds that the same result can be recovered later, elsewhere, on another machine. The first source of variation is the draw. At each step of inference, the model produces a probability for every possible fragment and draws one; temperature governs how spread out that distribution is, and at zero the most probable fragment wins systematically, which removes the draw as a source of randomness. That is not enough, because the distribution itself is not strictly stable: the computation is done in floating point, where addition is not associative, so that changing the order of operations changes the last digits of the result. And that order is not fixed by the request: it depends on how the computation is spread across the units of the machine, and above all on the make-up of the batch of requests processed together, which depends on the load on the service at that instant. It then takes only two fragments being almost equally probable for that tiny gap to tip the choice, and with it the whole remainder of the text, since each fragment produced feeds into the next.

The consequence for measurement

The heaviest consequence bears on measurement. A classic software test compares an output with an expected output; here the same input can give two different texts, both acceptable, so that line-by-line equality stops being a criterion. You therefore reason in rates over a batch of cases, replay each case several times, and look at the spread as much as at the average: an evaluation run on a single attempt does not measure a system, it measures a draw. The corollary is unpleasant and must be held to, since a gap of a few points between two versions may be nothing but noise, which only another run can settle. Debugging changes in nature too, since an incident cannot be reproduced at will: traces become the only record of what actually happened, and without them there is nothing left to examine. What gets stabilised is therefore not the output but what surrounds it, by imposing a structured output, validating the shape received, and making the harness deterministic where the model is not. A system can be reliable without being deterministic, and that is the only target genuinely within reach.

Stable is not right

The founding trap is to confuse stable with correct. A model is perfectly capable of being wrong in a perfectly reproducible way, and temperature at zero freezes the error exactly as it freezes the right answer. The second trap is the random seed, which people take to be enough: it fixes the randomness of the draw, not the order of the computations nor the make-up of the batches, and it survives neither a change of hardware nor an update to the service. The third is the version, since remotely served models evolve under the systems that use them: a result with no version noted beside it becomes unusable within months, and hosting an open weight model yourself settles that one point without settling the others. The fourth is contractual, and the most expensive: promising an identical output in a specification or in front of an auditor commits you to a property that nothing guarantees, whereas what can really be committed to is behaviour within bounds, measured on an evaluation set. It is worth noting that variation is not always a fault, since it is what allows several phrasings to be offered, a failed generation to be run again, or several answers to be drawn so as to keep only the one that comes back most often.