tokenisation

In depth

The tool and the operation

Two objects need separating, and confusing them muddles everything else: the tokeniser is the tool, tokenisation is the operation. The tokeniser has two pieces, a vocabulary of fragments and a splitting algorithm. The vocabulary is built before the model is trained, from a corpus of text: you start from single characters, repeatedly merge the most frequent pairs, and stop at a vocabulary size fixed in advance. Tokenising a text then means applying that table from left to right, keeping the longest available match, and replacing each fragment by its number. The operation is exactly reversible, the sequence of integers translating back character for character, which sets it clearly apart from the embedding, which loses the letter in order to keep the meaning. It works in both directions: at each step of generation, the model picks a fragment from that same vocabulary, never a word and never a letter.

The size of the vocabulary

The size of the vocabulary is a trade-off. A large one produces long fragments, therefore short sequences and less computation per text, at the price of a heavier input and output table to carry. A narrow one does the reverse. The make-up of the corpus used to build it then decides the cost of each language: whatever was rare in it gets split more finely, for identical content, so that the same sentence is not worth the same number of tokens from one language to another. Two practical effects follow, and they compound: the context window fills up faster, and the cost per token is paid on more tokens. Digits are a case apart, their splitting weighing enough on arithmetic that they are now handled in regular groups rather than as frequent sequences. Since the vocabulary belongs to the model and is frozen with it, changing model changes every count: a budget established on one does not transfer to another.

The trap

The first trap is the one this entry exists to clear: looking for a tokenisation setting where a chunking decision needs revisiting. The second concerns character-level tasks, counting the letters of a word, spelling it backwards, judging a rhyme: the word arrives as one indivisible piece, and the letters composing it are not directly visible. The third is counting by eye, which goes most wrong exactly where the stakes are highest: identifiers, addresses, code and tables come close to one token per character, and only a counter for the model in question gives a usable figure. The fourth is silent truncation, since a text that exceeds the window is not refused but cut, and it is almost always a rough estimate that let it get there. The last is a security matter: invisible characters or visually identical variants produce different sequences of fragments, which makes it possible to have the model read something other than what a person sees on screen.