关于基于解码器的预训练语言模型的多语言能力：发现并控制语言特定神经元
On the Multilingual Ability of Decoder-based Pre-trained Language Models: Finding and Controlling Language-Specific Neurons
AI总结：
本文研究多语言解码器PLM的语言处理机制，通过六种语言识别出低重叠、集中于首尾层的语言特定神经元，并用少于1%的神经元篡改显著控制生成语言概率。
摘要：
当前基于解码器的预训练语言模型（PLM）成功展现出多语言能力。然而，这些模型如何处理多语言仍不清楚。我们分析多语言、基于解码器的PLM在神经元层面的内部行为，具体考察仅解码器多语言PLM内部是否存在“唯一为每种语言”激活的神经元。我们分析六种语言：英语、德语、法语、西班牙语、中文和日语，并表明语言特定神经元具有唯一性，不同语言之间仅有轻微重叠（低于5%）。这些神经元主要分布在模型的最初几层和最后几层。该趋势在不同语言和模型之间保持一致。此外，我们在推理期间篡改每个模型中不足总神经元1%的神经元，并证明篡改少数语言特定神经元会显著改变文本生成中目标语言出现的概率。
英文摘要：
Current decoder-based pre-trained language models (PLMs) successfully demonstrate multilingual capabilities. However, it is unclear how these models handle multilingualism. We analyze the neuron-level internal behavior of multilingual decoder-based PLMs, Specifically examining the existence of neurons that fire ``uniquely for each language'' within decoder-only multilingual PLMs. We analyze six languages: English, German, French, Spanish, Chinese, and Japanese, and show that language-specific neurons are unique, with a slight overlap (< 5%) between languages. These neurons are mainly distributed in the models' first and last few layers. This trend remains consistent across languages and models. Additionally, we tamper with less than 1% of the total neurons in each model during inference and demonstrate that tampering with a few language-specific neurons drastically changes the probability of target language occurrence in text generation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
