GlossaryFloor 1 · The Modela solid block on its own: the prediction machineFloor 1 · The Model
token
No. 006 · v2026-08FR: tokenA token is the small piece of text that the model handles: not a letter, not quite a word, rather a chunk of a common word. What you write reaches it cut up in this way, like a text read syllable by syllable.
What it is not
A token is not a word. The cutting follows the frequency of character sequences, not grammar: a common word fits in a single token, whereas a rare word, a proper noun or a reference take up several. That is why the same text does not cost the same number of tokens from one language to another, and why counting the words of a document tells you only roughly whether it will fit in the context window.
In depth
The tokeniser
The cutting is produced by a tokeniser, built at the same time as the model from its training texts: it keeps a fixed vocabulary of frequent fragments, and any incoming text is rewritten as a sequence of these fragments. The model therefore never sees letters or words, only fragment numbers, which explains its clumsiness on tasks about characters: counting the letters of a word or writing it backwards asks it to reason about a material it does not perceive directly. A token does not exist in the absolute: it exists only relative to the cutting of a given model, and two models do not count the same text in the same way.
The unit of account
The token is also the unit of account of the system: billing, usage limits and a good part of the response time are measured in tokens, on the input as on the output. This produces counter-intuitive effects. A heavily structured table can cost more than a paragraph saying the same thing, because the punctuation and the formatting are paid for. And languages less represented in the training data are cut more finely, so they consume more tokens for equal content. Reasoning in pages or in characters leads to underestimating a bill or exceeding a limit without understanding why.
Two confusions
Two confusions coexist and deserve to be separated. The first is linguistic: the token is not a word, and intuitions drawn from counting words travel badly, particularly for identifiers, addresses and figures, which are very costly in tokens. The second is one of vocabulary: in the software world, a token also means an authentication token, with no relation whatsoever to the cutting of text, and the same meeting can therefore use the word in two senses. One practical trap remains: a text that is too long is not politely refused, it is truncated, and what disappears disappears in silence.
Under the hood2 steps · the real shape of the objects
A tokeniser is a table of fragments and a cutting algorithm. Nothing more: no grammar, no meaning. Seeing it at work explains at a stroke why the model stumbles on letters, why your bill is surprising, and why a text that is too long disappears in silence.
- 01
What becomes of a sentence
The cutting replaces the text with a sequence of fragment numbers. The boundaries are neither the letters nor the words: they are the frequent chunks of the training corpus, and the space that precedes a word is part of the fragment.
encode("The cat sleeps.") // → ["The", " cat", " sleeps", "."] 4 fragments // → [791, 8415, 33056, 13] what the model receives encode("The cat dozes.") // → ["The", " cat", " doz", "es", "."] 5: a rare word breaks up // → [791, 8415, 76322, 288, 13] encode("antidisestablishmentarianism") // → ["anti", "dis", "establishment", "arian", "ism"] 5 for ONE word encode("A-4417") // → ["A", "-", "44", "17"] 4: an identifier costs a lot- " cat"
- The space belongs to the fragment. That is why the same word at the start and in the middle of a sentence is not the same token, and why sticking two words together changes the cutting of both.
- [791, ...]
- The model sees only these integers. No letters, no words: the “r”s of “barrel” exist nowhere in what it receives, only the number of the fragment that held them.
The trapThe numbers above are illustrative: every tokeniser has its own table, and one model’s integers mean nothing to another. An exact count is asked of the tokeniser you are targeting.
- 02
Why the bill is surprising
The token is the unit of account: billing, limits, and part of the response time. Since the cutting follows frequency in the training corpus, what is rare costs a lot, and what is laid out is paid for.
// same information, two presentations count("Revenue 2024: 1.2 M€") // ~10 tokens count('| Year | Revenue |\n|---|---|\n| 2024 | 1.2 M€ |') // ~24 tokens // the punctuation of the table is paid for, the meaning is the same // same sentence, two languages count("The meeting is postponed to Tuesday.") // ~7 tokens count("La réunion est reportée à mardi.") // ~11 tokens // a less represented language is cut more finely // and what costs the most is not what you would think count("f47ac10b-58cc-4372-a567-0e02b2c3d479") // ~25 tokens // an identifier: almost one token per characterThe trapThe sizing error costs more than the billing one: a corpus of identifiers or of code takes up two to three times the room of ordinary text of the same apparent length, and it is the window that overflows.
What variesThe table of fragments belongs to the model and changes with it: an exact count requires the tokeniser of the model in question, never a general rule, and the numbers above are worth only an order of magnitude. What does not vary: the cutting follows frequency and not grammar, the unit of billing is the fragment, and a text that exceeds the limit is truncated without warning.
Relations where the neighbours live
Check 3 questions · click your answer
Level 1 · Recognise
You paste in a document of a thousand words. How many tokens is that?
Level 2 · Distinguish
The same text translated costs more to process in French than in English. Why?
Level 2 · Distinguish
Why does a model struggle to count the letters of a word?
Lexigraph, "Token", v2026-08, https://www.lexigraph.org/en/token/, CC BY 4.0.