GlossaryFloor 1 · The Modela solid block on its own: the prediction machineFloor 1 · The Model
distillation
No. 063 · v2026-08FR: distillationDistillation consists in training a small model to imitate the answers of a large one, in order to obtain a far lighter object that keeps the essentials of the behaviour. Like a workshop manual drawn from the experience of a master craftsman: shorter, and sufficient for most cases.
What it is not
Distillation is not fine-tuning. Fine-tuning starts from a model and adapts it to a domain with your examples; distillation manufactures a smaller model by having it learn from a larger one. Nor is it a lossless compression: the model obtained is weaker than its master, generally on rare cases and long chains of reasoning, that is to say precisely where it is not noticed straight away.
In depth
The process
The procedure has become simpler with language models. The large model is made to produce a great number of answers, possibly with their reasoning steps, then the small model is trained on that corpus. It learns not from the world but from another model, which is very efficient: it inherits a behaviour already refined without going through a full pre-training, which is out of reach for almost everyone.
The economic appeal
The economic interest is considerable and explains the present landscape. A model ten times smaller costs roughly ten times less per request, answers faster, fits on a modest machine, sometimes on a phone. For a well-delimited task, classifying, extracting, rewording, the quality gap with the large model often becomes negligible. The right architecture is therefore not always to call the best model everywhere, but to reserve the large ones for the difficult cases.
Two limits
Two limits must be known. The first is legal: the terms of use of most services forbid using their outputs to train a competing model, and the procedure has already given rise to public accusations between vendors. The second is qualitative: the distilled model also inherits the errors and biases of its master, without the margin that allowed the master to recover. It must therefore be evaluated for itself, and not trusted on the grounds that it descends from a well-regarded model.
Relations where the neighbours live
Check 3 questions · click your answer
Level 1 · Recognise
What is the small model trained on, in a distillation?
Level 2 · Distinguish
Where does the quality loss of a distilled model show up first?
Level 2 · Distinguish
You want to distil a proprietary model into an internal one. What do you check?
Lexigraph, "Distillation", v2026-08, https://www.lexigraph.org/en/distillation/, CC BY 4.0.