Babel Tower的兴衰：探究多语言代码大型语言模型的演化过程
The Rise and Down of Babel Tower: Investigating the Evolution Process of Multilingual Code Large Language Model
AI总结：
本文以代码LLMs为平台探究多语言能力演化，提出Babel Tower Hypothesis揭示语言知识系统从共享到分化的发展过程，并据此构建优化预训练语料库，显著提升了模型多语言性能。
摘要：
大型语言模型（LLMs）展现出了显著的多语言能力。然而，在预训练阶段这些能力发展的潜在机制尚不清楚。本文以代码LLMs为实验平台，探索预训练过程中LLMs多语言能力的演化。基于观察，我们提出了Babel Tower Hypothesis，描述了LLMs获取新语言能力的整个过程。在学习过程中，多种语言最初共享一个由主导语言支配的单一知识系统，随后逐渐发展出特定语言的知识系统。随后，我们通过识别工作语言和语言转移神经元来追踪LLMs的内部状态，从而验证了上述假设。实验结果表明，LLM内部状态的变化与我们的Babel Tower Hypothesis一致。基于这些见解，我们提出了一种为多语言代码LLMs构建优化预训练语料库的新方法，该方法显著优于在原始语料库上训练的LLMs。提出的Babel Tower Hypothesis为设计预训练数据分布以实现LLMs最佳多语言能力提供了新见解。
英文摘要：
Large language models (LLMs) have shown significant multilingual capabilities. However, the mechanisms underlying the development of these capabilities during pre-training are not well understood. In this paper, we use code LLMs as an experimental platform to explore the evolution of multilingual capabilities in LLMs during the pre-training process. Based on our observations, we propose the Babel Tower Hypothesis, which describes the entire process of LLMs acquiring new language capabilities. During the learning process, multiple languages initially share a single knowledge system dominated by the primary language and gradually develop language-specific knowledge systems. We then validate the above hypothesis by tracking the internal states of the LLMs through identifying working languages and language transferring neurons. Experimental results show that the internal state changes of the LLM are consistent with our Babel Tower Hypothesis. Building on these insights, we propose a novel method to construct an optimized pre-training corpus for multilingual code LLMs, which significantly outperforms LLMs trained on the original corpus. The proposed Babel Tower Hypothesis provides new insights into designing pre-training data distributions to achieve optimal multilingual capabilities in LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
