PreAlign：通过早期建立多语言对齐来提升跨语言迁移
PreAlign: Boosting Cross-Lingual Transfer by Early Establishment of Multilingual Alignment
AI总结：
本文提出PreAlign框架，通过在预训练前初始化对齐词表示并采用代码切换策略，显著提升了大型语言模型的跨语言迁移和知识共享能力。
摘要：
大型语言模型尽管主要以英语为中心进行预训练，但展现出尚可的多语言能力。然而，这些模型中自发的多语言对齐较弱，导致跨语言迁移和知识共享不尽如人意。以往的工作试图通过在预训练期间或之后显式注入多语言对齐信息来解决此问题。因此，在预训练的早期阶段，跨语言共享信息或知识的对齐较弱。在本文中，我们提出了PreAlign，这是一个在语言模型预训练之前建立多语言对齐的框架。PreAlign通过初始化模型以生成对齐词的相似表示来注入多语言对齐，并在预训练期间使用代码切换策略来保持这种对齐。在合成的英语到英语克隆设置中的大量实验表明，PreAlign在语言建模、零样本跨语言迁移和跨语言知识应用方面显著优于标准的多语言联合训练。在真实场景中的进一步实验进一步验证了PreAlign在各种模型规模上的有效性。
英文摘要：
Large language models demonstrate reasonable multilingual abilities, despite predominantly English-centric pretraining. However, the spontaneous multilingual alignment in these models is shown to be weak, leading to unsatisfactory cross-lingual transfer and knowledge sharing. Previous works attempt to address this issue by explicitly injecting multilingual alignment information during or after pretraining. Thus for the early stage in pretraining, the alignment is weak for sharing information or knowledge across languages. In this paper, we propose PreAlign, a framework that establishes multilingual alignment prior to language model pretraining. PreAlign injects multilingual alignment by initializing the model to generate similar representations of aligned words and preserves this alignment using a code-switching strategy during pretraining. Extensive experiments in a synthetic English to English-Clone setting demonstrate that PreAlign significantly outperforms standard multilingual joint training in language modeling, zero-shot cross-lingual transfer, and cross-lingual knowledge application. Further experiments in real-world scenarios further validate PreAlign's effectiveness across various model sizes.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
