脚本税：衡量基于分词驱动的效率和延迟差异在多语言语言模型中的表现
The Script Tax: Measuring Tokenization-Driven Efficiency and Latency Disparities in Multilingual Language Models
AI总结：
本研究发现多语言语言模型中基于分词的效率和延迟差异显著，通过比较不同正字法变体，揭示了分词对不同书写系统的影响，并提出脚本意识的分词和预训练方法。
摘要：
预训练的多语言语言模型通常被假设为脚本无关，但它们的分词器可能对某些书写系统施加系统性的成本。我们通过比较两个具有相同语言内容的正字法变体来量化这种脚本税。在mBERT和XLM-R中，更高的碎片化正字法显示出生育率增加约3.4倍（6.73-6.85 vs. 2.10-2.35 tokens/word），导致在相同硬件上推理速度下降16.5倍（0.23 vs. 3.8 sentences/second）。使用每字符比特数（BPC）以避免子词碎片化的“NLL悖论”，我们发现信息成本显著增加：mBERT增加19.7%（8.06->9.65），XLM-R增加47.1%（12.19->17.94）。回程转换检查（CER_rt=0.31）表明这些差距反映的是正字法条件下的处理而非映射噪声。我们的结果突显了分词作为多语言NLP中不平等的关键来源，并推动了脚本意识的分词和预训练。
英文摘要：
Pretrained multilingual language models are often assumed to be script-agnostic, yet their tokenizers can impose systematic costs on certain writing systems. We quantify this script tax by comparing two orthographic variants with identical linguistic content. Across mBERT and XLM-R, the higher-fragmentation orthography shows a ~3.4x increase in fertility (6.73-6.85 vs. 2.10-2.35 tokens/word), leading to a 16.5x inference slowdown (0.23 vs. 3.8 sentences/second) on identical hardware. Using bits per character (BPC) to avoid the "NLL paradox" from subword fragmentation, we find a substantial increase in information cost: +19.7% for mBERT (8.06->9.65) and +47.1% for XLM-R (12.19->17.94). A round-trip conversion check (CER_rt=0.31) suggests these gaps reflect orthography-conditioned processing rather than mapping noise. Our results highlight tokenization as a key source of inequity in multilingual NLP and motivate script-aware tokenization and pretraining.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
