Tokenisation · Case, digits and spaces

Easy to miss

Small changes, different tokens

Text is cut into tokens before a model reads it, and which pieces you get is decided by how common that exact text is. That makes the count sensitive to things you’d never think of as meaningful: whether you shouted, how long a number is, whether there’s a space in front.