The token tax: why the same message costs more in some languages

What the token tax is

AI services measure text in tokens, the pieces a model cuts text into, and they charge per token. The catch is that the same message does not come to the same number of tokens in every language. Write it in English and it is cheap. Write the exact same meaning in Russian or Japanese and it can take more than twice as many. We call that gap the token tax; in the course you will find it under the heading "The bill, by language".

What the video shows

The video explains that AI splits text into pieces and charges per piece. The same paragraph took 31 pieces in English and 78 in Russian. It also brings good news: newer tokenizers are closing the gap.

The numbers

The course took one paragraph, translated it, and measured every version with two real tokenizers that OpenAI ships: the older one, cl100k, and the newer one, o200k.

| Language | Characters | Tokens (cl100k) | Tokens (o200k) | Extra vs English (cl100k) | |---|---|---|---|---| | English | 164 | 31 | 31 | none | | Spanish | 169 | 43 | 36 | +39% | | Russian | 178 | 78 | 43 | +152% | | Japanese | 72 | 79 | 58 | +155% |

Look at the Japanese row: 72 characters became 79 tokens. That is more tokens than characters.

Why it costs more, three times over

Three things are counted in tokens, so the tax is charged three times at once:

• Price. APIs, the paid connections that apps use to talk to a model, bill per token.
• Room. A model's context window, how much text it can keep in view at once, is measured in tokens, so less of your document fits.
• Speed. More tokens means a slower answer.

The cause is the training text. A tokenizer that learned mostly from English spent its shortcuts on English letter combinations. Spanish uses the same alphabet, so it gets some of the benefit. Russian letters take two bytes each in the standard way of storing text, and few of those byte pairs were common enough to earn a shortcut. Japanese needs three bytes per character, so it fares worst.

An everyday example

Imagine an old telegram office that charges per code group. Its code book was written by English speakers, so common English phrases have short codes. A message in another language has to be spelled out bit by bit, and it costs more for exactly the same meaning. Nobody set out to overcharge anyone. The code book was simply built from one language.

The good news

This is a problem that can be fixed, and it is being fixed. The newer o200k tokenizer doubled the vocabulary and rebalanced the text it learned from. That cut Spanish's extra cost from +39% to +16%, and Russian's from +152% to +39%. It is also the real reason AI vocabularies keep growing: not squeezing text for its own sake, but shrinking the extra that a large part of the world was quietly being charged.

Why it matters

If you build with AI in a language other than English, measure token counts in your own language before you set a budget or promise how much text will fit. And when you compare models, remember that the tokenizer is part of the price.

Learn it step by step in Chapter 7 of our free course AI From Scratch: Build a BPE Tokenizer: Why Your Model Can't Count the R's.