假朋友并非敌人：探究多语言语言模型中的词汇重叠
False Friends Are Not Foes: Investigating Vocabulary Overlap in Multilingual Language Models
AI总结：
本研究通过受控实验探究多语言模型中token重叠对跨语言迁移的影响，发现重叠（尤其是语义相似的重叠）能增强嵌入空间的跨语言语义关系并提升XNLI和XQuAD上的迁移性能，表明共享词汇是有益的设计选择。
摘要：
在多语言语料库上训练的子词分词器自然会在不同语言间产生重叠的token。token重叠是促进跨语言迁移，还是反而在语言间引入干扰？先前的研究提供了相互矛盾的证据，部分原因在于实验设置多样以及混淆因素（如token频率或子词分割粒度）的存在。为解决此问题，我们设计了一项受控实验，在系统变化词汇重叠设置的多种语言对上训练双语自回归模型。关键在于，我们探索了一个理解重叠如何影响迁移的新维度：跨语言共享token的语义相似性。我们首先分析模型的隐藏表示，发现任何类型的重叠都会创建捕捉跨语言语义关系的嵌入空间，而这种效应在词汇不相交的模型中要弱得多。在XNLI和XQuAD上，我们发现具有重叠词汇的模型优于词汇不相交的模型，且迁移性能通常随重叠增加而提升。总体而言，我们的研究结果凸显了token重叠在多语言模型中的优势，并表明大量共享词汇仍是多语言分词器有益的设计选择。
英文摘要：
Subword tokenizers trained on multilingual corpora naturally produce overlapping tokens across languages. Does token overlap facilitate cross-lingual transfer or instead introduce interference between languages? Prior work offers mixed evidence, partly due to varied setups and confounders, such as token frequency or subword segmentation granularity. To address this question, we devise a controlled experiment where we train bilingual autoregressive models on multiple language pairs under systematically varied vocabulary overlap settings. Crucially, we explore a new dimension to understanding how overlap affects transfer: the semantic similarity of tokens shared across languages. We first analyze our models' hidden representations and find that overlap of any kind creates embedding spaces that capture cross-lingual semantic relationships, while this effect is much weaker in models with disjoint vocabularies. On XNLI and XQuAD, we find that models with overlap outperform models with disjoint vocabularies, and that transfer performance generally improves as overlap increases. Overall, our findings highlight the advantages of token overlap in multilingual models and show that substantial shared vocabulary remains a beneficial design choice for multilingual tokenizers.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
