多语言语言模型如何记忆事实？
How Do Multilingual Language Models Remember Facts?
AI总结：
本文分析三个多语言LLM的事实回忆机制，发现既有英语机制基本可迁移，并揭示语言相关的客体提取及功能向量的两阶段信息组合。
摘要：
大型语言模型（LLM）存储并检索在预训练期间获得的大量事实知识。先前研究已经定位并识别了知识回忆背后的机制；然而，这些研究仅关注英语单语模型。这些机制如何推广到非英语语言和多语言LLM，这一问题仍未得到探索。本文中，我们通过对三个多语言LLM进行全面分析来填补这一空白。首先，我们表明，先前在英语中识别出的回忆机制在很大程度上适用于多语言语境，但会因语言和架构而存在细微差异。接着，通过修补中间表示，我们定位了语言在回忆过程中的作用，发现主体丰富化与语言无关，而客体提取则依赖语言。此外，我们发现最后一个token的表示充当功能向量（FV），同时编码查询语言和要从主体中提取的内容。而且，在仅解码器LLM中，FV在两个独立阶段组合这两类信息。这些洞见揭示了多语言LLM在回忆信息时的独特机制，凸显了需要专门为多语言LLM设计的新方法——例如知识评估、事实编辑和知识获取。
英文摘要：
Large Language Models (LLMs) store and retrieve vast amounts of factual knowledge acquired during pre-training. Prior research has localized and identified mechanisms behind knowledge recall; however, it has only focused on English monolingual models. The question of how these mechanisms generalize to non-English languages and multilingual LLMs remains unexplored. In this paper, we address this gap by conducting a comprehensive analysis of three multilingual LLMs. First, we show that previously identified recall mechanisms in English largely apply to multilingual contexts, with nuances based on language and architecture. Next, through patching intermediate representations, we localize the role of language during recall, finding that subject enrichment is language-independent, while object extraction is language-dependent. Additionally, we discover that the last token representation acts as a Function Vector (FV), encoding both the language of the query and the content to be extracted from the subject. Furthermore, in decoder-only LLMs, FVs compose these two pieces of information in two separate stages. These insights reveal unique mechanisms in multilingual LLMs for recalling information, highlighting the need for new methodologies -- such as knowledge evaluation, fact editing, and knowledge acquisition -- that are specifically tailored for multilingual LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
