From text to tokens

Why tokens?

A language model cannot manipulate text directly. It manipulates numbers. The very first step, every time you talk to a LLM, is to convert your text into a sequence of integers — the token IDs.

Tokenization is the splitting that makes this conversion possible.

Why not one token per word?

At first glance, you might imagine: one word = one token. Simple.

But it doesn't work:

• Language contains millions of possible words (inflected forms, neologisms, proper nouns, typos…). One token per word requires a gigantic vocabulary.
• The model can do nothing with a word it has never seen.
• Some languages (Chinese, Japanese) don't even have spaces between words.

The solution adopted by almost all modern LLMs: subwords.

Subword tokenization

With a subword tokenizer:

• Frequent words become a single token (the, and, is)
• Rare words are split into smaller pieces (tokenization → token + ization)
• Unknown characters can always be decomposed down to individual letters

The result: a vocabulary of reasonable size (typically from 30,000 to over 250,000 tokens — the range keeps creeping up, because recent tokenizers cover far more languages) that can represent any text.

The most widely used algorithm is called BPE (Byte Pair Encoding). It works by starting from individual characters and iteratively merging the most frequent pairs in the training corpus.

BPE in thirty seconds

Imagine a tiny corpus of three words: low, lower, lowest. Start by tokenizing at the character level:

l o w
l o w e r
l o w e s t

At each iteration, find the most frequent adjacent pair. Here, l o appears three times — merge it into lo:

lo w
lo w e r
lo w e s t

Now lo w is the most frequent pair. Merge it:

low
low e r
low e s t

Keep going until you reach the desired vocabulary size. Chunks that recur often (low) become a single token. Rare ones (est) stay decomposed. That's exactly what BPE does — across billions of words instead of three, and over bytes rather than characters in modern models (byte-level BPE), which guarantees nothing is ever "out of vocabulary".

A few cousins worth knowing: WordPiece (BERT) and Unigram LM (T5, mT5) are, like BPE, algorithms for building the vocabulary. Watch out for a common confusion: SentencePiece, usually listed alongside them, is not an algorithm but a library — it implements BPE and Unigram, and it is what T5, mT5 and the early Llamas (1 and 2) were built on. Since Llama 3 (2024), Meta has moved to a 128,000-token byte-level BPE, in the spirit of tiktoken. All of them share the same idea — a subword vocabulary — with different heuristics.

Special tokens

Beyond text subwords, the tokenizer reserves a few special tokens that never appear naturally:

• <|im_start|>, <|im_end|> (OpenAI), [INST]…[/INST] (Llama 2 and Mistral), <|start_header_id|>…<|eot_id|> (Llama 3 and later), <|user|> / <|assistant|> — delimit conversation turns. Each model family has its own format, and it changes from one generation to the next.
• <|endoftext|> — end of document.
• <|fim_prefix|>, <|fim_middle|> — for fill-in-the-middle, used in code completion.

These are what turn a dialogue "User said X, Assistant replied Y" into a single linear sequence of tokens the model can process. When you send a message to ChatGPT, these markers are added automatically before tokenization.

Try it

Subwords appear in the right pane: each token is a reusable fragment, not necessarily a complete word. Frequent words fit in a single token; rare ones break into several pieces.

A few things to notice:

• Short, frequent words are rarely split.
• Long or rare words often end up in multiple pieces.
• The space preceding a word is part of the token (that's why · hello and hello are different tokens).
• In English, the token/word ratio is typically lower than in other languages (1.2–1.4 versus 1.4–1.8 for French, on GPT-3/GPT-4-era tokenizers) — models have seen more English during training. Recent tokenizers, with their much larger and more multilingual vocabularies, have narrowed the gap. They have not closed it: other languages still cost more than English.

Practical consequences

This token business has lots of unexpected implications:

• LLMs count letters poorly. "How many R's in strawberry?" was the famous trap of 2024: plenty of models answered 2 instead of 3, because the word arrives in a few tokens, not as separate letters. Today's large models get it right — but the structural cause hasn't budged an inch. They have learned to work around it, by spelling the word out letter by letter before counting. Ask a recent model and watch its reasoning: that spelling-out step is often right there. It's the admission that direct counting still doesn't work.
• API prices are calculated in tokens, not words. Languages that tokenize less efficiently cost a bit more to process.
• Context windows (128k tokens, 200k tokens…) are also measured in tokens. A 100,000-word book represents roughly 130,000 tokens.

For the model, "tokenization" and "token·iza·tion" are the same thing. It only sees the pieces.

What's next

Now that your text has become a sequence of integers, those integers will enter the model. The first step inside the model: they are transformed into vectors, in a space of several hundred dimensions where meaning becomes a geometric position.

That's the subject of the next chapter.

Updated August 2, 2026