音素救援：基于国际音标的多语言分词
Phonemes to the Rescue: Multilingual Tokenization Based on International Phonetic Alphabet
AI总结：
提出使用国际音标（IPA）作为多语言分词的语言无关输入表示，通过更平衡的字节-字符分布和跨语言字符重叠，显著提升分词质量，尤其对非拉丁文字和未见语言更有效。
摘要：
多语言语言模型通常表现出跨语言的性能差异，这种差异可能早在分词阶段就出现。广泛使用的子词分词方法偏向高资源语言，而无分词器方法对于字节-字符比更高的文字仍会产生更长的序列。为了解决这些缺点，我们提出使用国际音标（IPA）作为多语言分词器的语言无关输入表示。IPA提供了紧凑的符号库存、更大的跨语言字符重叠以及跨语言更平衡的字节-字符分布。我们在24种语言和14种文字上训练了文本与IPA子词分词器的配对，并证明IPA分词器一致地提高了分词质量，特别是对于非拉丁文字，并且更有效地泛化到未见语言和文字。
英文摘要：
Multilingual language models often exhibit performance disparities across languages that can arise as early as the tokenization stage. Widely-used subword tokenization approaches favor high-resource languages, and tokenizer-free methods still yield longer sequences for scripts with a higher bytes-per-character ratio. To address these shortcomings, we propose to use the International Phonetic Alphabet (IPA) as a language-agnostic input representation for multilingual tokenizers. IPA provides a compact symbol inventory, greater cross-lingual character overlap, and a more balanced byte-per-character distribution across languages. We train matched pairs of text vs. IPA subword tokenizers across 24 languages and 14 scripts and demonstrate that IPA tokenizers consistently improve tokenization quality, especially for non-Latin scripts, and generalize more effectively to unseen languages and scripts.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
