arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.07231cs.CLcs.AI
收藏

EconCausal: 面向大语言模型的上下文感知经济推理基准

EconCausal: A Context-Aware Economic Reasoning Benchmark for Large Language Models

Graduate School of Data Science, KAIST(韩国科学技术院数据科学研究生院);College of Business, KAIST(韩国科学技术院商学院);Data Science for Humanity Group, MPI-SP(马克斯·普朗克所际数据科学为人类集团);School of Computing, KAIST(韩国科学技术院计算学院);Division of Social Science, HKUST(香港科技大学社会科学系)

Donggyu Lee, Hyeok Yun, Meeyoung Cha, Sungwon Park, Sangyoon Park, Jihee Kim

AI总结:

提出EconCausal基准,包含从顶级经济金融期刊提取的10,490个上下文标注因果三元组,评估大语言模型在指定上下文中推断因果方向及随上下文变化调整判断的能力。

摘要:

社会经济因果效应高度依赖于制度和环境背景。相同的干预措施在不同监管制度、市场条件、时间段或人群中可能产生不同甚至相反的效果。这对大语言模型(LLM)在决策支持角色中提出了挑战:它们能否在指定上下文中推断因果效应的方向,并在上下文变化时修正该判断?为此,我们引入了EconCausal,这是一个大规模基准,包含从顶级经济和金融期刊的2,595项高质量实证研究中提取的10,490个上下文标注因果三元组,通过严格的四阶段流程构建,包括多轮共识、上下文细化和多批评者过滤。跨模型实验表明,LLM往往无法根据上下文调整其预测。虽然顶级模型在固定、显式上下文中达到88%的准确率,但在需要跨上下文修正符号的情况下,准确率下降32.6个百分点(从73.9%降至41.3%),一旦引入误导性的符号证据,准确率降至50%以下。模型还过度承诺于方向性(+/-)符号,仅在13.8%的情况下识别出零效应,且在这些类别上校准不良。数据集和基准公开于 https://anonymous.4open.science/r/econcausal-benchmark-6F12。

英文摘要:

Socio-economic causal effects depend heavily on their institutional and environmental contexts. The same intervention can produce different, even opposite, effects across regulatory regimes, market conditions, time periods, or populations. This poses a challenge for large language models (LLMs) in decision-support roles: can they infer the direction of a causal effect under a specified context, and revise that judgment when the context changes? To address this, we introduce EconCausal, a large-scale benchmark of 10,490 context-annotated causal triplets extracted from 2,595 high-quality empirical studies in top-tier economics and finance journals, constructed through a rigorous four-stage pipeline with multi-run consensus, context refinement, and multi-critic filtering. Across 19 models, we observe source-sign over-transfer in cross-context prediction. While top models reach ~84% accuracy in single-context prediction, mean accuracy is 74.75% overall but drops to 44.27% when source and target signs disagree. On these cases, accuracy is 11.54 percentage points (pp) lower with the signed source example than with the target input alone. Mean accuracy is below 60% under corrupted source signs. Models also struggle to predict study-reported nonsignificant effects (19.16% accuracy). The dataset and benchmark are publicly available at https://anonymous.4open.science/r/econcausal-benchmark-6F12.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑