语言模型中跨语言泛化的体外研究
An In-Vitro Study on Cross-Lingual Generalization in Language Models
AI总结：
通过构建两种程序生成的语言，独立控制词汇距离、少数语言比例等变量，研究语言模型跨语言迁移的机制，发现迁移主要取决于分词是否保留可复用的跨语言子结构，且词汇量越小越有利于掩码迁移。
摘要：
在自然语料中，语言模型的跨语言迁移难以研究，因为词汇重叠、形态、数据不平衡和分词相互纠缠。我们引入了一个体外框架，使用两种程序生成的语言，它们共享相同的本体、类型化语法和组合结构，但表面实现不同。这使我们能够独立改变词汇距离、少数语言比例、分词器训练制度和词汇量大小，同时评估在掩码少数语言条件下的迁移，该条件的词汇形式在训练中从未被观察到。在700次受控运行中，我们发现迁移受分词器平衡或原始词汇相似性的影响较小，而更多地取决于分词是否保留可复用的跨语言子结构。较小的词汇量通常通过保持单词可分解为共享片段来改善掩码迁移，而较大的词汇量可能将形式转化为特定语言的原子。我们进一步表明，迁移是一个阶段性过程：语法和类型级能力先于掩码词汇泛化。最后，我们尝试通过分词器桥梁解释这一机制，并表明桥梁强度与掩码可达性密切相关。
英文摘要：
Cross-lingual transfer in language models is difficult to study in natural corpora because lexical overlap, morphology, data imbalance, and tokenization are entangled. We introduce an in-vitro framework with two procedurally generated languages that share the same ontology, typed grammar, and compositional structure, but differ in surface realization. This lets us independently vary lexical distance, minority-language proportion, tokenizer training regime, and vocabulary size, while evaluating transfer on a masked minority-language condition whose lexical forms are never observed during training. Across 700 controlled runs, we find that transfer is governed less by tokenizer balance or raw lexical similarity than by whether tokenization preserves reusable cross-lingual substructure. Smaller vocabularies often improve masked transfer by keeping words decomposable into shared fragments, whereas larger vocabularies can turn forms into language-specific atoms. We further show that transfer emerges as a staged process: grammatical and type-level competence precede masked lexical generalization. Finally, we attempt to explain this mechanism through tokenizer bridges and show that bridge strength correlates strongly with masked reachability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
