迷失于多语言性：剖析Transformer语言模型中的跨语言事实不一致性
Lost in Multilinguality: Dissecting Cross-lingual Factual Inconsistency in Transformer Language Models
AI总结：
本文通过机制可解释性方法揭示多语言语言模型跨语言事实不一致源于最后几层语言转换失败，并提出线性捷径方法绕过这些层以提升准确率与一致性。
摘要：
多语言语言模型（MLMs）在不同语言中存储事实知识，但在面对不同语言中语义等价的提示时，往往难以给出一致的回答。尽管已有研究指出了这一跨语言不一致问题，其深层原因仍未得到探索。本文采用机制可解释性方法研究MLMs中的跨语言不一致性。我们发现，MLMs在大部分网络层中以语言无关的概念空间编码知识，仅在最后几层才转换到语言特定的空间。语言转换过程中的失败通常会导致目标语言中的错误预测，即使该答案在其他语言中是正确的。为了缓解这种不一致问题，我们提出了一种线性捷径方法，绕过最后几层的计算，从而同时提升预测准确率和跨语言一致性。我们的发现揭示了MLMs的内部机制，并提供了一种轻量且有效的策略，用于生成更一致的事实性输出。
英文摘要：
Multilingual language models (MLMs) store factual knowledge across languages but often struggle to provide consistent responses to semantically equivalent prompts in different languages. While previous studies point out this cross-lingual inconsistency issue, the underlying causes remain unexplored. In this work, we use mechanistic interpretability methods to investigate cross-lingual inconsistencies in MLMs. We find that MLMs encode knowledge in a language-independent concept space through most layers, and only transition to language-specific spaces in the final layers. Failures during the language transition often result in incorrect predictions in the target language, even when the answers are correct in other languages. To mitigate this inconsistency issue, we propose a linear shortcut method that bypasses computations in the final layers, enhancing both prediction accuracy and cross-lingual consistency. Our findings shed light on the internal mechanisms of MLMs and provide a lightweight, effective strategy for producing more consistent factual outputs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
