Defeasibility in causal reasoning implies that the causal relationship between cause and effect can be strengthened or weakened. Namely, the causal strength between cause and effect should increase or decrease with the incorporation of strengthening arguments (supporters) or weakening arguments (defeaters), respectively. However, existing works ignore defeasibility in causal reasoning and fail to evaluate existing causal strength metrics in defeasible settings. In this work, we present $δ$-CAUSAL, the first benchmark dataset for studying defeasibility in causal reasoning. $δ$-CAUSAL includes around 11K events spanning ten domains, featuring defeasible causality pairs, i.e., cause-effect pairs accompanied by supporters and defeaters. We further show current causal strength metrics fail to reflect the change of causal strength with the incorporation of supporters or defeaters in $δ$-CAUSAL. To this end, we propose CESAR (Causal Embedding aSsociation with Attention Rating), a metric that measures causal strength based on token-level causal relationships. CESAR achieves a significant 69.7% relative improvement over existing metrics, increasing from 47.2% to 80.1% in capturing the causal strength change brought by supporters and defeaters. We further demonstrate even Large Language Models (LLMs) like GPT-3.5 still lag 4.5 and 10.7 points behind humans in generating supporters and defeaters, emphasizing the challenge posed by $δ$-CAUSAL.
Com$^2$: A Causal-Guided Benchmark for Exploring Complex Commonsense Reasoning in Large Language Models
Research Center for Social Computing and Interactive Robotics(社会计算与交互机器人研究中心);Harbin Institute of Technology(哈尔滨工业大学);Institute of Trustworthy Embodied AI(可信具身人工智能研究院);Fudan University(复旦大学);Beijing Academy of Artificial Intelligence(北京人工智能研究院);Nanjing University(南京大学)
Exploring and steering the moral compass of Large Language Models
Ernst Strüngmann Institute for Neuroscience(恩斯特·施特伦格曼神经科学研究所);Max Planck Society(马克斯·普朗克学会);Laboratory for Clinical Neuroscience(临床神经科学实验室);Centre for Biomedical Technology(生物医学技术中心);Universidad Politécnica de Madrid(马德里理工大学)
Designing Computational Tools for Exploring Causal Relationships in Qualitative Data
Department of Computer Science, National University of Singapore(国立新加坡大学计算机科学系);Wee Kim Wee School of Communication and Information, Nanyang Technological University(南洋理工大学Wee Kim Wee通讯与信息学院);Dept of Computer Science, National Yang Ming Chiao Tung University(阳明交通大学计算机科学系)
Exploring Generative Process Reward Modeling for Semi-Structured Data: A Case Study of Table Question Answering
University of Augsburg(奥格斯堡大学);Institut für Maschinelle Sprachverarbeitung, University of Stuttgart(斯图加特大学机械语言处理研究所);Bosch Center for Artificial Intelligence(博世人工智能中心)