Why the Hindi support queue costs three times more
In one line
The tokenizer was fitted to English, so every other language pays a surcharge in tokens, context and latency — a fairness problem hiding inside a cost line.
The situation
The assistant is rolled out to three new markets. Finance notices that the per-conversation cost in Hindi is nearly three times the English figure, Japanese is not far behind, and Spanish is up by a third. The conversations are no longer; the customers are not asking harder questions. Something in the pipeline is pricing languages differently, and nobody chose it.
The vocabulary has a native language#
A 's is built by merging the most frequent character sequences in its training corpus. If that corpus is mostly English, then English words and word-pieces earn single tokens, and everything else is represented by whatever fragments happen to exist. A common Hindi word can take five or six tokens where its English equivalent takes one. Japanese, with no spaces and a large character set, fares similarly.
The same sentence — "your order has shipped and will arrive on Tuesday" — might be 12 tokens in English, 16 in Spanish, 30 in Japanese and 36 in Hindi. Every downstream cost is counted in those tokens, so the surcharge is not a rounding error; it is the whole bill.
Newer tokenizers with larger, more multilingual vocabularies narrow the gap considerably. It rarely closes entirely, and it is a property of the model you picked, not a setting you can change.
It is not only money#
Three things scale with token count, and cost is the least important.
The fills three times faster. A conversation that comfortably fits its history and retrieved documents in English is truncating in Hindi, so the assistant forgets earlier turns sooner and sees fewer supporting passages. Quality degrades for exactly the users who are already paying more.
rises with prompt length, so the same interface feels slower. And output limits set in tokens — "keep answers under 300 tokens" — become answers under 100 words in Hindi, which reads as curt.
Which means this is a product and fairness question wearing a cost label. The fixes are the same ones you would apply anywhere: measure token counts per language rather than assuming, set budgets in tokens per language rather than globally, trim the parts of the prompt that carry no signal, and — when choosing a model — put multilingual tokenizer efficiency on the evaluation sheet next to accuracy.
Questions to ask
- Have I measured tokens per message by language, or assumed parity?
- Are my context, output and cost budgets set globally in tokens? Which languages hit them first?
- Does the model I chose have a tokenizer built for the languages I serve?
- Is quality being measured per language, or only in the language the team speaks?
Mindset
The tokenizer quietly encodes whose text the model was built for. Anything you measure in tokens — cost, context, latency, output length — inherits that bias, and it only shows up if you break the numbers down by language.
Where it connects
Got the shape of it?
Examples do not count toward phase progress — that stays on the topics. This is just so the list remembers what you have seen.