CruxBench：信息发现基准

中文摘要 AI 辅助

大型语言模型（LLM）的基准测试通常根据固定参考答案来评估答案的准确性。然而，在许多复杂的现实世界任务中，一个核心步骤是首先确定哪些问题值得提出：将难题分解为子问题——我们称之为“关键问题”（cruxes）——其答案为解决目标问题提供关键步骤。为了评估这种信息发现能力，我们引入了CruxBench，这是一个通过信息价值（VOI）来评估LLM生成的问题的基准：即模型提出的关键问题在多大程度上更新了关于目标预测问题的信念。CruxBench罕见地结合了三个关键特性：（1）构造上具有抗污染性，因为真实答案由未来世界事件生成；（2）开放式，允许无限制且复杂的基于文本的提交，而非单一正确的数值答案；（3）有依据性，信息性根据现实世界信念的量化变化来衡量。我们在293个目标预测问题上评估了八个多样化的模型，发现VOI与模型能力的独立度量高度相关（r=0.90），并捕捉了关键问题对于回答目标问题的有用性。然而，即使对于前沿LLM，信息发现仍然具有挑战性，它们仅略微优于随机时间基线。

英文摘要

Benchmarks for large language models (LLMs) typically evaluate the accuracy of answers against fixed reference labels. But a central step in many complex real-world tasks is identifying which questions are worth asking in the first place: decomposing a difficult problem into subquestions -- which we call cruxes -- whose answers provide key steps on the path toward solving the target problem. To evaluate this capability of information discovery, we introduce CruxBench, a benchmark that grades LLM-generated questions by their Value of Information (VOI): how much a model-proposed crux updates beliefs about a target forecasting question. CruxBench enjoys a rare combination of three key properties: it is (1) contamination-resistant by construction, since ground truth is generated by future world events; (2) open-ended, admitting unbounded and complex text-based submissions rather than one correct numeric answer; and (3) grounded, with informativeness measured against quantified changes in real-world beliefs. We evaluate a diverse set of eight models on 293 target forecasting questions and find that VOI correlates highly with independent measures of model capability (r=0.90) and captures cruxes' usefulness for answering target questions. However, information discovery remains challenging even for frontier LLMs, which only narrowly outperform a random-timing baseline.

发表机构

• The University of Chicago（芝加哥大学）
• Forecasting Research Institute（预测研究所）
• Federal Reserve Bank of Chicago（芝加哥联邦储备银行）

机构由 AI 辅助整理，请以论文原文为准。

补充信息

备注

NeurIPS 2026