非洲语言税：量化前沿大语言模型中标记化非洲语言的成本、延迟和上下文惩罚
The African Language Tax: Quantifying the Cost, Latency, and Context Penalty of Tokenizing African Languages in Frontier LLMs
AI总结：
本研究通过平行语料库测量20种非洲语言在11种前沿分词器上的标记化惩罚，发现非洲语言平均标记数比英语高1.88倍（N'Ko语最高达8.92倍），导致推理成本增加8.9倍、有效上下文窗口仅剩11%，并发布开源测量工具和缓解指南。
摘要：
商业大语言模型按标记计费、缩放延迟和预算上下文。然而，分词器为某些语言中相同含义分配的子词标记多于其他语言，因此具有高标记生育率的语言使用者在使用模型之前就面临结构性惩罚。这种惩罚在一般多语言环境中已有记录，但尚未在企业部署经济学和认知上下文容量层面系统测量非洲语言。我们使用平行语料库测量了20种非洲语言（涵盖5个语系和3种文字：拉丁、吉兹/埃塞俄比亚、N'Ko；其中19种出现在主要FLORES-200+语料库中，尼日利亚皮钦语仅通过MAFAND-MT测量），从而隔离语言效应与内容。在FLORES-200+上的11种前沿和开放分词器中，每种非洲语言相对于英语都有标记化溢价（GPT-5 / o200k_base中位数为1.88倍，N'Ko最高达8.92倍）；惩罚最大的是埃塞俄比亚和N'Ko文字（达到7-9倍），且在不同语料库间几乎不变（FLORES与SIB-200的Pearson r = 0.9998）。转化为部署术语，这导致高达8.9倍的推理成本和等效的生成延迟倍数（GPT-5上N'Ko vs英语；阿姆哈拉语为7.4倍），以及仅相当于英语11%的有效上下文窗口。目前对非洲语言最好的分词器Gemma 4将平均溢价从3.31倍（cl100k_base）降至2.38倍，但没有分词器消除惩罚。我们发布了一个开源测量工具（afri-fertility）、一个公开排行榜、一个结果数据集以及针对非洲开发者的缓解指南。惩罚最严重地落在那些最负担不起的语言使用者身上，这是一种直接编码到子词词汇中的数字鸿沟。
英文摘要：
Commercial large language models bill, scale latency, and budget context per token. Yet tokenizers assign more subword tokens to the same meaning in some languages than in others, so speakers of languages with high token-fertility pay a structural penalty before a model is ever invoked. This penalty is documented for multilingual settings in general, but it has not been measured systematically for African languages at the level of enterprise deployment economics and cognitive context capacity. We measure it across 20 African languages spanning five language families and three scripts (Latin, Ge'ez/Ethiopic, N'Ko; 19 appear in the primary FLORES-200+ corpus, with Nigerian Pidgin measured via MAFAND-MT only), using parallel corpora so that the language effect is isolated from content. Across 11 frontier and open tokenizers on FLORES-200+, every African language carries a tokenization premium above English (median 1.88x on GPT-5 / o200k_base, up to 8.92x for N'Ko); the penalty is largest for Ethiopic and N'Ko scripts (reaching 7-9x) and is near-invariant across corpora (FLORES vs SIB-200 Pearson r = 0.9998). Translated into deployment terms, this results in up to 8.9x inference cost and an equivalent generation-latency multiplier (N'Ko vs English on GPT-5; 7.4x for Amharic), and as little as 11% of English's effective context window. The best currently available tokenizer for African languages, Gemma 4, reduces the mean premium from 3.31x (cl100k_base) to 2.38x, but no tokenizer eliminates the penalty. We release an open measurement tool (afri-fertility), a public leaderboard, a results dataset, and mitigation guidance for African builders. The penalty falls hardest on the languages whose speakers can least afford it, a digital divide encoded directly into the subword vocabulary.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
