CRAB：评估现实世界事件间因果关系的强度
CRAB: Assessing the Strength of Causal Relationships Between Real-world Events
AI总结：
本研究提出用于评估现实叙事事件因果理解能力的新基准CRAB，含2700对事件细粒度标注，测试发现大语言模型在该任务表现不佳，复杂因果结构下推理能力更弱，相关资源已开源。
摘要：
理解叙事需要对文本提及的事件间的因果关系进行推理。尽管现有基础模型在众多需要推理的NLP任务中取得了亮眼成果，但尚不清楚它们是否理解叙事中事件潜在因果关系网络的复杂性。本研究中，我们提出CRAB（因果推理评估基准），这是一款全新的基准测试工具，旨在评估模型对现实叙事中事件的因果理解能力。CRAB包含约2700对现实世界事件的细粒度上下文因果标注，这些事件描述了各类具有新闻价值的事件时间线（例如埃隆·马斯克收购Twitter事件）。借助CRAB，我们测评了多个大语言模型的性能，结果显示大多数系统在该任务上表现不佳。受经典因果原理启发，我们还分析了CRAB中事件组的因果结构，发现与简单线性因果链中的事件相比，当事件来自复杂因果结构时，模型的因果推理表现更差。我们已向研究界开放数据集和代码。
英文摘要：
Understanding narratives requires reasoning about the cause-and-effect relationships between events mentioned in the text. While existing foundation models yield impressive results in many NLP tasks requiring reasoning, it is unclear whether they understand the complexity of the underlying network of causal relationships of events in narratives. In this work, we present CRAB, a new Causal Reasoning Assessment Benchmark designed to evaluate causal understanding of events in real-world narratives. CRAB contains fine-grained, contextual causality annotations for ~2.7K pairs of real-world events that describe various newsworthy event timelines (e.g., the acquisition of Twitter by Elon Musk). Using CRAB, we measure the performance of several large language models, demonstrating that most systems achieve poor performance on the task. Motivated by classical causal principles, we also analyze the causal structures of groups of events in CRAB, and find that models perform worse on causal reasoning when events are derived from complex causal structures compared to simple linear causal chains. We make our dataset and code available to the research community.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
