mOthello：多语言模型中的跨语言表示对齐与跨语言迁移何时出现？
mOthello: When Do Cross-Lingual Representation Alignment and Cross-Lingual Transfer Emerge in Multilingual Models?
AI总结：
本研究提出合成任务mOthello，探究多语言模型中跨语言表示对齐与迁移的条件，发现锚定标记有助于对齐但语言无关表示不足，进而提出统一输出空间的多语言预训练方法以促进跨语言迁移。
摘要：
许多预训练的多语言模型展现出跨语言迁移能力，这通常被归因于在预训练期间学习到的语言无关表示。然而，目前尚不清楚哪些因素有助于学习语言无关表示，以及学习到的语言无关表示是否足以促进跨语言迁移。我们提出了一个合成任务——多语言奥赛罗（mOthello），作为探究这两个问题的测试平台。我们发现：（1）使用朴素多语言预训练训练的模型无法在所有输入语言上学习到语言无关表示；（2）引入“锚定标记”（即跨语言相同的词汇项）有助于跨语言表示对齐；（3）仅学习语言无关表示本身不足以促进跨语言迁移。基于我们的发现，我们提出了一种新方法——统一输出空间的多语言预训练——该方法既能诱导语言无关表示的学习，又能促进跨语言迁移。
英文摘要：
Many pretrained multilingual models exhibit cross-lingual transfer ability, which is often attributed to a learned language-neutral representation during pretraining. However, it remains unclear what factors contribute to the learning of a language-neutral representation, and whether the learned language-neutral representation suffices to facilitate cross-lingual transfer. We propose a synthetic task, Multilingual Othello (mOthello), as a testbed to delve into these two questions. We find that: (1) models trained with naive multilingual pretraining fail to learn a language-neutral representation across all input languages; (2) the introduction of "anchor tokens" (i.e., lexical items that are identical across languages) helps cross-lingual representation alignment; and (3) the learning of a language-neutral representation alone is not sufficient to facilitate cross-lingual transfer. Based on our findings, we propose a novel approach - multilingual pretraining with unified output space - that both induces the learning of language-neutral representation and facilitates cross-lingual transfer.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
