arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.03183cs.CL
收藏

探索因果推理中的可废止性

Exploring Defeasibility in Causal Reasoning

EPFL(洛桑联邦理工学院)

Shaobo Cui, Lazar Milikic, Yiyang Feng, Mete Ismayilzada, Debjit Paul, Antoine Bosselut, Boi Faltings

AI总结:

本文提出首个因果推理可废止性基准数据集$δ$-CAUSAL,并设计了基于token级因果关系的指标CESAR,相比现有指标相对提升69.7%,同时揭示GPT-3.5在生成支持者和反驳者方面仍落后人类。

摘要:

因果推理中的可废止性意味着因果关系中的因果关系可以被加强或削弱。即,因果强度应随着加强论据(支持者)或削弱论据(反驳者)的加入而分别增加或减少。然而,现有研究忽略了因果推理中的可废止性,且未能在可废止设置下评估现有的因果强度指标。在本研究中,我们提出了$δ$-CAUSAL,这是首个用于研究因果推理中可废止性的基准数据集。$δ$-CAUSAL包含约11K个事件,涵盖十个领域,具有可废止因果关系对,即伴随支持者和反驳者的因果对。我们进一步表明,当前的因果强度指标无法反映在$δ$-CAUSAL中加入支持者或反驳者后因果强度的变化。为此,我们提出了CESAR(基于注意力评分的因果嵌入关联),这是一种基于token级因果关系测量因果强度的指标。CESAR相比现有指标实现了显著的69.7%相对提升,在捕捉支持者和反驳者带来的因果强度变化方面,从47.2%提高到80.1%。我们进一步证明,即使是像GPT-3.5这样的大型语言模型(LLM),在生成支持者和反驳者方面仍分别落后人类4.5和10.7个百分点,凸显了$δ$-CAUSAL带来的挑战。

英文摘要:

Defeasibility in causal reasoning implies that the causal relationship between cause and effect can be strengthened or weakened. Namely, the causal strength between cause and effect should increase or decrease with the incorporation of strengthening arguments (supporters) or weakening arguments (defeaters), respectively. However, existing works ignore defeasibility in causal reasoning and fail to evaluate existing causal strength metrics in defeasible settings. In this work, we present $δ$-CAUSAL, the first benchmark dataset for studying defeasibility in causal reasoning. $δ$-CAUSAL includes around 11K events spanning ten domains, featuring defeasible causality pairs, i.e., cause-effect pairs accompanied by supporters and defeaters. We further show current causal strength metrics fail to reflect the change of causal strength with the incorporation of supporters or defeaters in $δ$-CAUSAL. To this end, we propose CESAR (Causal Embedding aSsociation with Attention Rating), a metric that measures causal strength based on token-level causal relationships. CESAR achieves a significant 69.7% relative improvement over existing metrics, increasing from 47.2% to 80.1% in capturing the causal strength change brought by supporters and defeaters. We further demonstrate even Large Language Models (LLMs) like GPT-3.5 still lag 4.5 and 10.7 points behind humans in generating supporters and defeaters, emphasizing the challenge posed by $δ$-CAUSAL.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑