超越英语中心的大语言模型：多语言模型用哪种语言思考？
Beyond English-Centric LLMs: What Language Do Multilingual Language Models Think in?
AI总结：
本研究揭示非英语中心LLM（如Swallow和LLM-jp）具有日语和英语双潜在语言，并随层变化，加深了对多语言模型内部语言动态的理解。
摘要：
在本研究中，我们探讨了非英语中心的大语言模型（LLM）尽管性能强劲，是否在其各自的主导语言中进行“思考”；更准确地说，“思考”指的是中间层的表示在解嵌入到词汇空间后，在生成过程中对某些主导语言表现出更高的概率。我们将此类语言称为内部潜在语言（latent language）。我们考察了三类典型的日语处理模型的潜在语言：Llama2，一个以英语为中心的模型；Swallow，一个以英语为中心但继续用日语预训练的模型；以及LLM-jp，一个在均衡的英语和日语语料库上预训练的模型。我们的实证发现表明，与仅依赖英语作为内部潜在语言的Llama2不同，针对日语的Swallow和LLM-jp同时使用日语和英语，展现出双重内部潜在语言。对于任何给定的目标语言，模型会优先激活与其最相关的潜在语言。此外，我们探讨了中间层如何响应涉及潜在内部语言与目标输出语言之间文化冲突的问题。我们还进一步研究了语言身份在层间如何转变，同时保持中间层表示中反映的一致语义含义。这项研究加深了对非英语中心大语言模型的理解，突显了其中间层内语言表示的复杂动态。
英文摘要：
In this study, we investigate whether non-English-centric LLMs, despite their strong performance, `think' in their respective dominant language: more precisely, `think' refers to how the representations of intermediate layers, when un-embedded into the vocabulary space, exhibit higher probabilities for certain dominant languages during generation. We term such languages as internal $\textbf{latent languages}$. We examine the latent language of three typical categories of models for Japanese processing: Llama2, an English-centric model; Swallow, an English-centric model with continued pre-training in Japanese; and LLM-jp, a model pre-trained on balanced English and Japanese corpora. Our empirical findings reveal that, unlike Llama2 which relies exclusively on English as the internal latent language, Japanese-specific Swallow and LLM-jp employ both Japanese and English, exhibiting dual internal latent languages. For any given target language, the model preferentially activates the latent language most closely related to it. In addition, we explore how intermediate layers respond to questions involving cultural conflicts between latent internal and target output languages. We further explore how the language identity shifts across layers while keeping consistent semantic meaning reflected in the intermediate layer representations. This study deepens the understanding of non-English-centric large language models, highlighting the intricate dynamics of language representation within their intermediate layers.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
