因果推理基准:用于解耦评估因果识别和估计的现实世界基准
CausalReasoningBenchmark: A Real-World Benchmark for Disentangled Evaluation of Causal Identification and Estimation
AI总结:
本文提出CausalReasoningBenchmark,通过173个查询和132个现实数据集评估因果识别与估计,揭示模型在研究设计细节上的瓶颈。
摘要:
许多自动化因果推断基准仅通过单一数值输出(如平均治疗效应ATE)评估系统性能,混淆了因果分析的两个步骤:识别(在给定假设下制定有效研究设计)和估计(在有限数据上实现该设计)。我们引入CausalReasoningBenchmark,包含173个查询和132个现实世界数据集,源自79篇同行评审论文和三本广泛使用的因果推理教科书。每个查询要求系统生成(i)结构化的识别规范,包括策略、处理、结果、控制变量及设计特定元素,以及(ii)带有标准误差的点估计。通过分别评分这两个组件,该基准可实现细粒度诊断:区分因果推理失败与数值执行错误。基于最先进的LLM的基准测试结果表明,尽管模型在79%的情况下正确识别了高层策略,但完整识别规范的正确性仅降至34%,揭示瓶颈在于研究设计的细微差别而非计算。CausalReasoningBenchmark已在Hugging Face上公开,旨在促进更稳健的自动化因果推理系统的发展。
英文摘要:
Many benchmarks for automated causal inference evaluate a system's performance based on a single numerical output, such as an Average Treatment Effect (ATE). This approach conflates two distinct steps in causal analysis: identification, formulating a valid research design under stated assumptions, and estimation, implementing that design numerically on finite data. We introduce CausalReasoningBenchmark, a benchmark of 174 queries across 139 real-world datasets, curated from 73 peer-reviewed research papers and three widely used causal-inference textbooks. For each query, a system must produce (i) a structured identification specification that names the strategy, the treatment, outcome, and control variables, and all design-specific elements, and (ii) a point estimate with a standard error. By scoring these two components separately, our benchmark enables granular diagnosis: it distinguishes failures in causal reasoning from errors in numerical execution. Baseline results with a state-of-the-art LLM show that, while the model correctly identifies the high-level strategy in 81% of cases, full identification-specification correctness drops to only 38%, revealing that the bottleneck lies in the nuanced details of research design rather than in computation. CausalReasoningBenchmark is publicly available on Hugging Face and is designed to foster the development of more robust automated causal-inference systems.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。