AI language taxation
How context windows and token use can vary between written languages.
AI Tokenizer is "taxing" non-English languages
I have a geeky interest, no surprise. Having worked with multi-lingual web development, and multi-lingual "paper" flyer/brochure development over the years, you can't help but get a little interested in the physical nature of the written word and how it fills physical space (or screen space) on the "page". I stumbled on a something interesting adjacent to this yesterday.
AI LLM use is currently billed in tokens, I have other posts covering the detail of that. Interestingly, the same sentences do not necessarily generate/consume the same number of tokens in every written language. English is represented relatively efficiently by many commonly used tokenizers, but other languages can be chopped up into many more pieces to represent exactly the same thing. This results in a differing number of tokens for the same content.
This is not just of geeky interest as it can ultimately end up in higher invoices for your overseas AI projects, let me explain...
AI language tax
This effect is sometimes called the "AI language tax".
Let us imagine two documents:
- one written in English
- one translated into another language
-identical except for the language they are written in.
When tokienized it can be found that one document will consume a lot more tokens than the other. I've worked for decades with multi-lingual content for business reports and websites, knowing all too well that different languages consume different amounts of page space, and this feels very familiar to that, even though it is not the same cause.
As an aside, to indulge myself, here is a table comparing common Western languages by roughly how much page space they tend to take up relative to English:
| Language | Relative Space vs. English | Notes |
|---|---|---|
| English | 100% (baseline) | Reference point |
| Spanish | ~115–125% | Longer words, more articles/prepositions |
| French | ~115–120% | Similar expansion to Spanish |
| Italian | ~110–120% | Slightly more compact than French/Spanish |
| Portuguese | ~110–120% | Comparable to Spanish |
| German | ~130–140% | Long compound nouns drive expansion |
| Dutch | ~115–130% | Compounding similar to German, less extreme |
| Swedish | ~105–115% | Fairly compact for a Germanic language |
| Norwegian | ~105–115% | Similar to Swedish |
| Danish | ~105–115% | Similar to Swedish/Norwegian |
| Finnish | ~120–130% | Agglutinative, long words |
| Polish | ~110–125% | Longer words, complex inflection |
| Greek | ~110–120% | Slightly longer than English on average |
When we tokenise we get a similar effect but not entirely from the language length.
Cost implications
Because the words get split up into tokens differently between languages, often producing more tokens in languages other than English, this naturally means more tokens will be processed, then by extension higher invoiced token use when working with other languages.
How the same amount of language can consume very different numbers of tokens
| Language | Rough tokens per word* | Compared with English |
|---|---|---|
| English | 1.2 | baseline |
| French / Spanish / Italian | 1.5–1.7 | ~25–40% more |
| Germanic languages | 1.7–1.9 | ~40–60% more |
| Slavic languages | 2.2–2.5 | ~80–110% more |
| Finnish / Baltic languages | 2.7–3.0 | ~125–150% more |
| Greek | ~3.1 | ~160% more |
*Illustrative results from a 2026 study of parallel European-language text across ten model tokenizers. The exact figure varies by tokenizer, model and content
I am struck by just how large the difference can be between languages. This is simply a side effect of how tokenizers work.
This can really matter for your AI projects...
Small scale users are not going to notice this bloat effect. However if you're developing and operating AI services across a large number of customer interactions, you probably should be aware of it. For, say you were processing millions of words a month, processing support requests from a help desk, this difference is going to get significant to you. When the English conversations tokenize efficiently, but another language requires many more tokens to represent similar content, resulting in processing for those customers can costing significantly more.
From my other blog posts, remember we pay both ways, the model consumes tokens while reading the customer's message and then consumes more while writing the answer. You could even have reasoning tokens on top of that too. This really mounts up when paying for API use.
This is why you may find a system costing more to run for one region of the world office vs another, the written language counts.
Context windows will be affected too
Context windows are fixed capacity. If a language requires more tokens to represent the same amount of text as the English version, you guessed it, then less information fits inside the same context window for that other language vs English. Something very obvious when you think about it, but again could explain why one language is running out of context vs another.
Language tax now results in:
higher processing cost + less useful context capacity -ouch!
Why do tokenizers bloat other written languages?
Text patterns that are common and well represented can often be encoded efficiently, whilst less well represented words, character combinations or writing systems may need breaking into smaller pieces. -Those extra pieces mean extra tokens. Also remember that different tokenizers result in different numbers of tokens.
As AI training data is very biased towards English language due to the training material being English biased, the other language fragments are simply not as frequently encountered by the tokenizers during AI training. Sub-words, such as, "ing", "tion", "the" get their own single token because they appear so frequently in English, however less well represented languages do not get the same dedication of tokens to the language parts that make up their language. This results in those words/language constructs having to be broken up into even smaller parts that do have token representation as there is no token in existence to represent them efficiently, as English has claimed those efficient tokens. I'm sure this is over simplifying it but it is good enough to convey the concept for the layperson.
Non-Latin, such as Chinese, Japanese Hindi, Russian use different scripts entirely, if the tokeniser has not learned to encode these, it will fall back to just character encoding, even in worst case individual characters, which is horrific for token volume.
Languages with heavy inflection or agglutination (yeah I am not a linguist either bear with it), Finnish, Turkish, German, Polish, Hungarian) build long words out of many meaningful parts. Since the tokenizer wasn't trained with dedicated tokens for every possible combination, it breaks these into several pieces.
Diacritics, we know the pain of them, from our Enterprise IT, where we have had to debug many integrations with "blocky" characters. Diacritics and accented characters can fragment tokens too. These characters can cause token fragmentation around the diacritical marks, if the tokeniser has not been designed around and so is not treating these text fragments as single units of text.
Other things that come into play around this:
- written languages
- writing systems
- tokenizer designs differ between models and AI provider companies
- model generations
- model cache
- the type of content you are processing
There are many factors involved, so its not simple to predict the "tax" that will be incurred.
Summary
If you're comparing the cost of AI services between countries or languages, cost per million tokens can be particularly misleading without taking a more holistic view of things
A deployment to a German speaking company, might appear to be consuming more AI than an English speaking company. It is simply paying more tokens to represent a similar volume of human communication.
Tokens are a great unit of measure for a computers to count and meter, however they are not the unit the business should be trying to optimise. The language differences are important to be aware of when designing systems that are multi-lingual and interface LLMs.
Ultimately its just geeky interesting too.