公平与效率：多语言大语言模型分词器的实证研究
Equity with Efficiency: An Empirical Study of Tokenizers for Multilingual Large Language Models
AI总结：
本文系统比较了11种东南亚语言上的公平分词器，发现Parity-aware BPE在效率与公平的权衡中处于帕累托前沿，而Morphology-Driven Byte Encoding在语义推理上表现最佳。
摘要：
多语言大语言模型（LLMs）依赖子词分词来桥接离散文本和连续神经表示。最先进的多语言LLMs通常使用字节级字节对编码（BPE）分词器，这些分词器在结构上偏向高资源语言和拉丁文字。对于代表性不足的语言使用者，特别是东南亚地区的语言，这种偏见增加了推理成本并扩大了跨语言能力差距。我们首次在涵盖11种东南亚语言的统一基准上对公平分词器进行了系统比较。除了分词器级别的压缩效率和跨语言公平性分析外，我们还通过使用相同训练数据训练的1.5B参数语言模型评估了下游任务性能。我们的结果表明，Parity-aware BPE位于效率-公平权衡的帕累托前沿，以有竞争力的成本实现了强大的压缩公平性。Morphology-Driven Byte Encoding通过形态更丰富的表示提供了最佳的语义推理性能，尽管计算成本更高。Byte Latent Transformer在下游任务中表现不佳，可能是因为其架构假设与有限低资源训练数据的约束不一致。总之，我们的发现表明跨语言公平性和分词效率并非根本矛盾，并为设计公平的多语言模型提供了实用指导。
英文摘要：
Multilingual large language models (LLMs) depend on subword tokenization to bridge discrete text and continuous neural representation. State-of-the-art multilingual LLMs often use Byte-level Byte-Pair Encoding (BPE) tokenizers that structurally favor high-resource languages and Latin scripts. For speakers of underrepresented languages, particularly those across Southeast Asia, this bias inflates inference costs and widens cross-lingual capability gaps. We present the first systematic comparison of equitable tokenizers on a unified benchmark spanning 11 Southeast Asian languages. Beyond tokenizer-level analysis of compression efficiency and cross-lingual equity, we assess downstream task performance through controlled 1.5B-parameter language model training using the same training data. Our results show that Parity-aware BPE lies on the Pareto frontier of the efficiency-equity trade-off, achieving strong compression parity at competitive cost. Morphology-Driven Byte Encoding delivers the best semantic reasoning performance through morphologically richer representations, albeit at a higher computational expense. Byte Latent Transformer underperforms on downstream tasks, possibly because its architectural assumptions misalign with the constraints of limited low-resource training data. Together, our findings demonstrate that cross-lingual fairness and tokenization efficiency are not fundamentally at odds, and offer practical guidance for designing equitable multilingual models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
