arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.20094cs.AI
收藏

CausalFlip: 超越语义匹配的LLM因果判断基准

CausalFlip: A Benchmark for LLM Causal Judgment Beyond Semantic Matching

Yuzhe Wang, Yaochen Zhu, Jundong Li

AI总结:

提出CausalFlip基准,通过构建语义相似但因果答案相反的问题对和噪声前缀评估,揭示LLM依赖语义匹配而非因果推理,并验证内化因果推理方法可提升因果基础。

摘要:

随着大型语言模型(LLM)在复杂、高风险决策场景中的部署日益增加,将其推理建立在因果关系而非虚假相关性上变得至关重要。然而,在传统推理基准上的强性能并不能保证LLM真正的因果推理能力,因为高准确率可能源于记忆语义模式而非分析潜在的真实因果结构。为弥合这一关键差距,我们提出一个新的因果推理基准CausalFlip,旨在鼓励开发新的LLM范式或训练算法,使LLM推理基于因果关系而非语义相关性。CausalFlip由基于事件三元组构建的因果判断问题组成,这些三元组可形成不同的混杂、链和碰撞器关系。在此基础上,对于每个事件三元组,我们构建语义相似的问题对,这些问题对重用相同的事件但产生相反的因果答案,使得严重依赖语义匹配的模型被系统性地推向错误预测。为进一步探究模型对语义模式的依赖,我们引入噪声前缀评估,在中间因果推理步骤前添加因果无关的文本,而不改变潜在的因果关系或推理过程的逻辑。我们在多种训练范式下评估LLM,包括仅答案训练、显式思维链(CoT)监督以及一种提出的内化因果推理方法,该方法旨在减轻推理过程中对相关性的显式依赖。我们的结果表明,显式CoT仍可能被虚假的语义相关性误导,而内化推理步骤则显著改善了因果基础,这表明更好地激发基础LLM的潜在因果推理能力是有前景的。

英文摘要:

As large language models (LLMs) witness increasing deployment in complex, high-stakes decision-making scenarios, it becomes imperative to ground their reasoning in causality rather than spurious correlations. However, strong performance on traditional reasoning benchmarks does not guarantee true causal reasoning ability of LLMs, as high accuracy may still arise from memorizing semantic patterns instead of analyzing the underlying true causal structures. To bridge this critical gap, we propose a new causal reasoning benchmark, CausalFlip, designed to encourage the development of new LLM paradigm or training algorithms that ground LLM reasoning in causality rather than semantic correlation. CausalFlip consists of causal judgment questions built over event triples that could form different confounder, chain, and collider relations. Based on this, for each event triple, we construct pairs of semantically similar questions that reuse the same events but yield opposite causal answers, where models that rely heavily on semantic matching are systematically driven toward incorrect predictions. To further probe models' reliance on semantic patterns, we introduce a noisy-prefix evaluation that prepends causally irrelevant text before intermediate causal reasoning steps without altering the underlying causal relations or the logic of the reasoning process. We evaluate LLMs under multiple training paradigms, including answer-only training, explicit Chain-of-Thought (CoT) supervision, and a proposed internalized causal reasoning approach that aims to mitigate explicit reliance on correlation in the reasoning process. Our results show that explicit CoT can still be misled by spurious semantic correlations, where internalizing reasoning steps yields substantially improved causal grounding, suggesting that it is promising to better elicit the latent causal reasoning capabilities of base LLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑