Tokenisation · What tokens cost

No such thing as unknown

Underneath, it’s just bytes

Text reaches a model as tokens, and a tokeniser never meets text it can’t handle. When a character is too rare to have earned a token, it falls back to the raw bytes that spell it out. A strawberry emoji is four bytes of UTF-8, and it arrives as two tokens — neither of which is a whole character on its own.