arXiv 2604.115022026-04-16cs.CLcs.AI

METER：评估大型语言模型中的多级上下文因果推理

METER: Evaluating Multi-Level Contextual Causal Reasoning in Large Language Models

College of Computer Science, Sichuan University（四川大学计算机学院）；Institute of Data Science, National University of Singapore（新加坡国立大学数据科学研究所）；Engineering Research Center of Machine Learning and Industry Intelligence, Ministry of Education, China（中国教育部机器学习与产业智能工程研究中心）

Pengfeng Li, Chen Huang, Chaoqun Hao, Hongyao Chen, Xiao-Yong Wei, Wenqiang Lei, See-Kiong Ng

备注：: ACL 2026. Our code and dataset are available at https://github.com/SCUNLP/METER

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出METER基准，系统评估大型语言模型在因果阶梯的三个层级上的表现，发现随着任务层级升高，模型能力显著下降，揭示了模型在因果推理中的两大失效模式。

摘要：

本文提出METER基准，系统评估大型语言模型在因果阶梯的三个层级上的表现，发现随着任务层级升高，模型能力显著下降，揭示了模型在因果推理中的两大失效模式：一是模型容易被因果无关但事实正确的信息所干扰；二是随着任务层级上升，对提供上下文的忠实度下降，导致性能降低。

英文摘要：

Contextual causal reasoning is a critical yet challenging capability for Large Language Models (LLMs). Existing benchmarks, however, often evaluate this skill in fragmented settings, failing to ensure context consistency or cover the full causal hierarchy. To address this, we pioneer METER to systematically benchmark LLMs across all three levels of the causal ladder under a unified context setting. Our extensive evaluation of various LLMs reveals a significant decline in proficiency as tasks ascend the causal hierarchy. To diagnose this degradation, we conduct a deep mechanistic analysis via both error pattern identification and internal information flow tracing. Our analysis reveals two primary failure modes: (1) LLMs are susceptible to distraction by causally irrelevant but factually correct information at lower level of causality; and (2) as tasks ascend the causal hierarchy, faithfulness to the provided context degrades, leading to a reduced performance. We belive our work advances our understanding of the mechanisms behind LLM contextual causal reasoning and establishes a critical foundation for future research. Our code and dataset are available at https://github.com/SCUNLP/METER .

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。