arXiv 2606.309872026-07-01cs.CLecon.GNq-fin.EC

自然语言解释中的判断质量测量：来自预测锦标赛的证据

Measuring Judgment Quality in Natural-Language Explanations: Evidence from Forecasting Tournaments

Forecasting Research Institute（预测研究所）；Good Judgment Inc（Good Judgment公司）；University of Toronto（多伦多大学）；Vector Institute for Artificial Intelligence（向量人工智能研究所）；Stanford University（斯坦福大学）；School of Arts and Sciences & Wharton, University of Pennsylvania（宾夕法尼亚大学文理学院与沃顿商学院）；Federal Reserve Bank of Chicago（芝加哥联邦储备银行）；Federal Reserve System（联邦储备系统）

Christopher W. Karvetski, Sheldon S. Huang, Simas Kučinskas, Nadja Flechner, Jingyu Hu, Philip Tetlock, Ezra Karger

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出解释质量标记（EQMs），用大语言模型评分60种理论驱动的推理模式，在超过55,000个预测-理由对中预测准确性，优于传统方法。

摘要：

决策者通常依赖专家判断及其书面解释，但解释质量难以大规模测量。预测锦标赛提供了一个天然测试场：概率判断与自然语言理由配对，并根据实际结果评分。我们引入了解释质量标记（EQMs），这是一组由理论驱动的六十种推理模式，由大语言模型（LLMs）评分。在一项预注册分析中，我们对来自多年预测锦标赛的超过55,000个预测-理由对进行了分析，EQMs在预测层面和预测者层面均能预测准确性，始终优于LLM之前的文本分析方法。超过90%的统计显著的模式级EQM-准确性相关性与我们的方向性假设一致。信号是不对称的：EQMs在识别可能表现不佳者方面比区分最优秀的预测者更可靠。与传统预测技能指标相比，EQMs在预测层面是最强的预测因子，在预测者层面具有竞争力，但弱于先前的准确性。人类对理由质量的评分与准确性的相关性较低，且过分重视理由长度。结果可迁移至一项独立的预测研究。EQMs提供了一种可扩展、可解释的方法，用于从书面解释中提取与判断相关的信息。

英文摘要：

Decision-makers routinely rely on expert judgments accompanied by written explanations, yet explanation quality is difficult to measure at scale. Forecasting tournaments offer a natural testing ground: probabilistic judgments are paired with natural-language rationales and scored against realized outcomes. We introduce Explanation Quality Markers (EQMs), a set of sixty theory-guided reasoning patterns scored by large language models (LLMs). In a pre-registered analysis of over 55,000 forecast-rationale pairs from a multiyear forecasting tournament, EQMs predict accuracy at both the forecast and forecaster levels, consistently outperforming pre-LLM text-analysis methods. More than 90% of statistically significant pattern-level EQM-accuracy correlations match our directional hypotheses. The signal is asymmetric: EQMs identify likely underperformers more reliably than they distinguish the very best forecasters. Benchmarked against traditional indicators of forecasting skill, EQMs are the strongest predictor at the forecast level and competitive at the forecaster level, though weaker than prior accuracy. Human ratings of rationale quality are less consistently correlated with accuracy and place disproportionate weight on rationale length. Results transfer to an independent forecasting study. EQMs provide a scalable, interpretable method for extracting judgment-relevant information from written explanations.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。