Token
tokenisation · tokenizer
Kurz gesagt
A token is the smallest unit of text a language model processes: a common word fragment produced by a tokeniser. In English one token averages roughly four characters, or about 0.75 words. Context limits and API pricing are both counted in tokens, not words.
Models do not see letters or words. Text is first split by a tokeniser into fragments that appeared frequently in its training data. Common words become single tokens; rare words break into pieces; whitespace and punctuation carry their own.
"unbelievable" → "un" + "believ" + "able"
"the" → "the"
" " → one token per indent level in codeThree consequences follow, and all three cost money.
Languages are not equal. A tokeniser trained mostly on English fragments Korean or Japanese more aggressively. The same document costs more in tokens — and therefore more in currency and more of the context window — in those languages.
Character-level tasks are hard. A model asked to count letters in a word is working with fragments, not letters. This is why spelling puzzles fail in ways that look absurd next to the model's other abilities.
Output length is a cost lever. Since output tokens are the expensive side, "answer in three bullets" is a budget decision as much as a formatting one.
Häufige Fragen
- How many tokens is 1,000 words?
- Roughly 1,300 tokens in English. Korean, Japanese and Chinese text typically produces substantially more tokens for the same visible length, because those scripts fragment more under common tokenisers.
- Why am I billed for input and output separately?
- Input can be processed in parallel across the sequence; output must be generated one token at a time, each requiring a full pass through the model. Output therefore costs several times more per token.