GlossaryFloor 1 · The Modela solid block on its own: the prediction machineFloor 1 · The Model
pre-training
No. 052 · v2026-08FR: pré-entraînementPre-training is the first phase, the longest and the most expensive: the model reads immense quantities of text, learning only to predict what comes next. Like years of reading that build a general culture, before anyone teaches you to answer a question politely.
What it is not
Pre-training is not fine-tuning: the first builds a model from nothing and is counted in months of computation across thousands of machines, the second reworks an existing model and can be done in a few hours. Nor is it what makes a model usable: as it comes out of pre-training, a model continues texts without knowing how to follow an instruction, refuse a request or stop. Almost nobody ever interacts with a model that has only been pre-trained.
In depth
The task
The task is disconcertingly simple given what it produces: hide the continuation of a text and ask the model to guess it, billions of times. No labels are needed, which makes it possible to use considerable corpora with no annotation work. It is this mechanism that builds, as a side effect, grammar, factual regularities, styles and part of the reasoning: to predict the continuation of a piece of reasoning well, you have to have captured something of its shape.
Post-training
Then comes a tuning phase, often called post-training, which changes the nature of the object without changing its knowledge. The model is taught to follow instructions from examples written for that purpose, then to prefer certain answers to others, from comparisons judged by humans or by other models. It is this second phase that produces the assistant: the tone, the refusal, the structure of the answers come from it, not from pre-training.
What the separation explains
Separating these two phases explains a great deal of behaviour. A model is unaware of events later than its corpus because it is pre-training that sets the cut-off date, and no subsequent tuning fills that gap. Conversely, two models issued from the same pre-training can differ profoundly in the way they answer, since everything that concerns behaviour is installed afterwards. It is also what explains why most organisations do not pre-train: they start from an existing model and work on the floor above.
Relations where the neighbours live
Check 3 questions · click your answer
Level 1 · Recognise
What can a model do coming out of pre-training alone?
Level 2 · Distinguish
Does a model updated in March know about events from February of the same year?
Level 2 · Distinguish
Where does an assistant’s tone, polite and structured, mainly come from?
Who works with this 2 roles
The roles for which this term is part of the ordinary work.
Lexigraph, "Pre-training", v2026-08, https://www.lexigraph.org/en/pre-training/, CC BY 4.0.