探究大语言模型训练过程中跨语言对齐的涌现
Probing the Emergence of Cross-lingual Alignment during LLM Training
AI总结：
本研究利用内在探测技术分析多语言LLM BLOOM的训练检查点，发现跨语言神经元重叠度与零样本跨语言迁移性能高度相关，且预训练中存在多语言能力下降阶段，揭示了跨语言对齐的涌现规律。
摘要：
多语言大语言模型（LLM）在零样本跨语言迁移性能上表现出色。我们推测，这一能力的基础是模型无需平行句对的显式监督即可实现语言对齐。尽管已知收敛后不同语言中互为翻译的句子的表征具有相似性，但目前尚不清楚这种跨语言对齐在LLM预训练过程中是如何涌现的。本研究利用内在探测技术（该技术可识别编码语言特征的神经元子集），将跨语言神经元重叠程度与给定模型的零样本跨语言迁移性能建立关联。具体而言，我们采用了多语言自回归LLM BLOOM在不同训练步骤和模型规模下的检查点开展研究。我们观察到神经元重叠度与下游任务性能之间存在高度相关性，这支持了我们关于有效跨语言迁移实现条件的假设。值得注意的是，我们还发现在预训练过程的某些阶段，隐式对齐能力和多语言能力均会出现下降，这为理解多语言预训练的动态过程提供了新的见解。
英文摘要：
Multilingual Large Language Models (LLMs) achieve remarkable levels of zero-shot cross-lingual transfer performance. We speculate that this is predicated on their ability to align languages without explicit supervision from parallel sentences. While representations of translationally equivalent sentences in different languages are known to be similar after convergence, however, it remains unclear how such cross-lingual alignment emerges during pre-training of LLMs. Our study leverages intrinsic probing techniques, which identify which subsets of neurons encode linguistic features, to correlate the degree of cross-lingual neuron overlap with the zero-shot cross-lingual transfer performance for a given model. In particular, we rely on checkpoints of BLOOM, a multilingual autoregressive LLM, across different training steps and model scales. We observe a high correlation between neuron overlap and downstream performance, which supports our hypothesis on the conditions leading to effective cross-lingual transfer. Interestingly, we also detect a degradation of both implicit alignment and multilingual abilities in certain phases of the pre-training process, providing new insights into the multilingual pretraining dynamics.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
