TokenorWord
The token is the unit the model handles and the one you pay for: a frequent fragment of text, which does not follow grammar. The word is a unit of meaning, and it has no existence for the model.
Point by point
| Criterion | Floor 1Floor 1 · The Modela solid block on its own: the prediction machineToken | Word |
|---|---|---|
| Who uses it | The model, and your invoice | Humans |
| What defines it | How often a run of characters occurs | Grammar and meaning |
| Stability | Depends on the model used | Independent of any machine |
| A rare word | Costs several tokens | Is still one word |
| Across languages | Varies widely for the same content | Does not vary |
| Used to measure | Cost, limits, context window | The length of a text, for a reader |
On the ground four situations
You assume that a thousand-word document will fit inside a thousand-unit limit.
WordYou reasoned in words. The splitting will produce more than that, and the margin you thought you had does not exist.
The same text translated into French costs more to process than it did in English.
Floor 1 · The Modela solid block on its own: the prediction machineTokenThe rate per token is identical: it is the splitting that differs, less well represented languages being broken into finer fragments.
A model gets the letters of a word wrong when asked to count them.
Floor 1 · The Modela solid block on its own: the prediction machineTokenThe useful counter-example: the word reaches it as one or two indivisible fragments. The letters are not visible to it, and no amount of reasoning will make them appear.
You write a short, dense instruction rather than an explanatory paragraph.
BothYou worked on the meaning, hence on words, and cut the bill, hence the tokens. The two units meet here, but they do not measure the same thing.
The test that settles it
Take an everyday word and an unusual proper noun, and run both through a token counter for the model you are targeting. If “water” costs one token while “Bishkek” costs three or four on its own, you have just seen that the model’s unit is not the word.
The trap
Converting pages into tokens with a single ratio. The ratio depends on the language, the model and the content: identifiers, addresses and figures cost far more than ordinary prose.
Check click your answer
Level 1 · Recognise
Does one word correspond to one token?
Level 2 · Distinguish
Why is the number of tokens in a text not a property of the text?
Level 2 · Distinguish
To check that a document will fit inside a context window, what do you do?