大型语言模型能否学习独立因果机制？
Can Large Language Models Learn Independent Causal Mechanisms?
AI总结：
本文提出由多个稀疏交互语言建模模块构成的新型 LLM 架构，通过引入独立因果机制约束提升分布外的抽象与因果推理性能，并分析机制独立性及领域不变性。
摘要：
尽管大型语言模型（LLM）在语言建模和复杂推理任务上表现出色，但在不常见场景或存在分布偏移的相同任务中却表现不足，显示出泛化能力的缺乏。相比之下，学习抽象变量和因果关系的因果模型等系统，能够对分布变化表现出更强的鲁棒性。这一成功的一个原因在于独立因果机制（ICM）的存在与使用，这些机制表示仅稀疏交互的高层概念。在本工作中，我们应用因果关系中的两个概念来学习 LLM 内部的 ICM。我们开发了一种新的 LLM 架构，由多个稀疏交互的语言建模模块组成。我们表明，此类因果约束可以提升抽象推理和因果推理任务上的分布外性能。我们还研究了独立性和领域专门化程度，并表明 LLM 依赖于预训练的、部分领域不变且对微调具有韧性的机制。
英文摘要：
Despite impressive performance on language modelling and complex reasoning tasks, Large Language Models (LLMs) fall short on the same tasks in uncommon settings or with distribution shifts, exhibiting a lack of generalisation ability. By contrast, systems such as causal models, that learn abstract variables and causal relationships, can demonstrate increased robustness against changes in the distribution. One reason for this success is the existence and use of Independent Causal Mechanisms (ICMs) representing high-level concepts that only sparsely interact. In this work, we apply two concepts from causality to learn ICMs within LLMs. We develop a new LLM architecture composed of multiple sparsely interacting language modelling modules. We show that such causal constraints can improve out-of-distribution performance on abstract and causal reasoning tasks. We also investigate the level of independence and domain specialisation and show that LLMs rely on pre-trained partially domain-invariant mechanisms resilient to fine-tuning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
