Tokenisation · What a token is

The unit of reading

A model never sees your words

Before anything else happens, your text is cut into tokens. Tokens are the only pieces a model can read. Some are whole words. Many are fragments.