Tokenisation · Why words split

Frequency decides

Common words stay whole

A model reads text as tokens — chunks that are sometimes whole words and often fragments. The vocabulary was built by counting: text that appears constantly earned its own token, and anything rarer has to be assembled from smaller pieces. What matters is how often a word shows up in writing, not how long it is.