AI language taxation

AI Tokenizer is "taxing" non-English languages

I found this subject interesting when I stumbled on it the other day.

The same amount of information does not necessarily generate/consume the same number of tokens in every language.

English is represented relatively efficiently by many commonly used tokenizers, but other languages can be chopped up into many more pieces to represent exactly the same thing.

This effect is sometimes referred to as the AI language tax.

Let us imagine two documents, each containing the same information:

• one written in English
• one translated into another language

Essentially identical except for the language they are written in.
One can consume significantly more tokens than the other. Anyone that works with multi-lingual reports and websites will know different languages consume different amounts of page space, this feels very similar.

Cost implications

Because the words get split up into tokens differently, producing more tokens in other languages, this naturally means more tokens will be processed, by extension, more costs will be incurred.

How the same amount of language can consume very different numbers of tokens

*Illustrative results from a 2026 study of parallel European-language text across ten model tokenizers. The exact figure varies by tokenizer, model and conten

It is striking how large the difference can be between languages.

It is simply a side effect of how tokenizers work.

Why this can matter

Small scale users are not going to notice this. However if you're operating AI services across a large number of customer interactions, you probably should.
If you were processing millions of words a month this is going to get significant.

If the English conversations tokenize efficiently, but another language requires twice as many tokens to represent similar content, then processing those customers can cost significantly more.

Be aware we pay both ways, the model consumes tokens while reading the customer's message and then consumes more while writing the answer. This really mounts up when paying for API use.

Context windows
If a language requires more tokens to represent the same amount of text as English, less actual information fits inside the same context window. Something very obvious when you think about it.

So the language tax increases to:

higher processing cost + less useful context capacity

This is getting painful!

Why?

Text patterns that are common and well represented can often be encoded efficiently.

Less well-represented words, character combinations or writing systems may need breaking into smaller pieces.

Those extra pieces mean extra tokens.

Other things that influence this:

• languages
• writing systems
• tokenizer designs
• model generations
• the type of content you are processing

This means there is not standard rule to apply so to figure out the "tax" applied by language, as there are many factors involved.

If you're comparing the cost of AI services between countries or languages, cost per million tokens can be particularly misleading.

A German company in your corporate group, might appear to be consuming more AI than your English operation when, in reality, it is simply paying more tokens to represent a similar amount of human communication.

Tokens are a great unit of measure for a comput to count, they are not the unit the business should be trying to optimise. The language differences are important to be aware of when designing systems that are multi-lingual that interface with AI LLMs.