A language model does not read letters or words. It reads tokens, chunks of text a few characters long, and every price, rate limit and memory ceiling is counted in them. Type anything below and four tokenizers chop it in front of you. They rarely agree.
cooking · work in progressEach tokenizer is a vocabulary, a list of chunks it knows, plus a rule for splitting text into them. The first two load on their own. The other two are a click away. The download is the vocabulary itself, which is why a bigger one costs more megabytes.
Every coloured chip is one token. A dashed chip is a single character that took more than one token, which happens when a vocabulary has never seen the letter and has to spell it out in bytes.
When the box holds more than one line, each line is counted on its own. Green is the cheapest line for that tokenizer, red the dearest.
Prices are quoted per million tokens, which sounds enormous and is not. Set a price and see what this text costs, once and a thousand times over. The dial goes from the cheapest small models to the dearest large ones; prices move often, so set it to whatever you are actually paying.
A vocabulary is built from the text a model was trained on. Common chunks get one token each, rare chunks are stitched from smaller pieces, and anything the vocabulary has never seen falls back to raw bytes, three of them per character for most non-Latin scripts. So the bill for a sentence depends less on the sentence than on what the tokenizer grew up reading.
That is why English is nearly always the cheapest language. Try the five-language preset: on the 2019 vocabulary the Hindi line costs about seven times the English one, on the newest about four. The difference is entirely down to how much of each language was in the training set when the vocabulary was built.
Watch a phone number. One tokenizer chunks the digits unpredictably, another insists on one token per digit. The second looks wasteful and is deliberate: a model that always sees digits one at a time is far better at arithmetic than one that sees 303 as a single unit one day and 30 plus 3 the next. Rare surnames are spelt out in fragments for the same reason common words are not, and one of the four lowercases everything before it starts, so it cannot tell a name from a noun.
A short paragraph is a few hundred tokens. A conversation with a model carries its whole history every turn, so a long one costs its entire length again each time you speak. The counter above says how many copies of your text fit in a million; for most texts it is thousands, not millions, and for a long document it is dozens.
These four are published as files anyone may download, and they span the algorithms and vocabulary sizes in use. Some commercial models publish their tokenizer and some do not, so the exact count a given service will bill you is only knowable from that service. The pattern is the same everywhere; the numbers differ by a little.
Counted, never judged. Nothing here says which tokenizer is better; a bigger vocabulary spends fewer tokens and more megabytes, and a model has to learn every entry in it.