揭示大型语言模型中的语言区域
Unveiling Linguistic Regions in Large Language Models
AI总结：
本文从区域划分角度研究LLMs跨语言对齐机制，发现约占1%参数的核心语言区域及各语言独立区域，并证明冻结核心区域可缓解进一步预训练中的灾难性遗忘问题。
摘要：
大型语言模型（LLMs）展现出了显著的跨语言对齐和泛化能力。当前的研究主要集中在提升LLMs的跨语言泛化能力上。然而，关于LLMs如何实现跨语言对齐的内在机制，目前仍缺乏研究。从区域划分的角度，本文对LLMs的语言能力进行了多项研究。我们在LLMs中发现了一个与语言能力相对应的核心区域，约占模型总参数的1%。通过将参数设为零来移除该核心区域，会导致30种不同语言的性能显著下降。此外，该核心区域表现出显著的维度依赖性，对特定维度上哪怕单个参数的扰动都会导致语言能力的丧失。而且，我们发现不同语言存在不同的单语区域，对这些特定区域的破坏会大幅降低LLMs在相应语言上的熟练度。我们的研究还表明，在进一步预训练期间冻结核心语言区域可以缓解灾难性遗忘（CF）问题，这是LLMs进一步预训练时观察到的常见现象。总体而言，探索LLMs的功能区域为其智能的基础提供了见解。
英文摘要：
Large Language Models (LLMs) have demonstrated considerable cross-lingual alignment and generalization ability. Current research primarily focuses on improving LLMs' cross-lingual generalization capabilities. However, there is still a lack of research on the intrinsic mechanisms of how LLMs achieve cross-lingual alignment. From the perspective of region partitioning, this paper conducts several investigations on the linguistic competence of LLMs. We discover a core region in LLMs that corresponds to linguistic competence, accounting for approximately 1% of the total model parameters. Removing this core region by setting parameters to zero results in a significant performance decrease across 30 different languages. Furthermore, this core region exhibits significant dimensional dependence, perturbations to even a single parameter on specific dimensions leading to a loss of linguistic competence. Moreover, we discover that distinct monolingual regions exist for different languages, and disruption to these specific regions substantially reduces the LLMs' proficiency in those corresponding languages. Our research also indicates that freezing the core linguistic region during further pre-training can mitigate the issue of catastrophic forgetting (CF), a common phenomenon observed during further pre-training of LLMs. Overall, exploring the LLMs' functional regions provides insights into the foundation of their intelligence.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
