证明时间：评估科学思想判断的基准
Proof of Time: A Benchmark for Evaluating Scientific Idea Judgments
AI总结：
PoT通过半可验证的基准框架评估科学思想判断，利用时间分区和未来可验证的目标，结合离线沙盒实现可扩展的评估。
摘要：
大语言模型越来越多地被用来评估和预测研究想法，但我们需要可扩展的方法来评估模型对这些科学想法的判断质量。为此，我们引入了PoT，一个半可验证的基准框架，将科学思想判断与后来可观察的下游信号（例如引用和研究人员议程的变化）联系起来。PoT冻结了一个预截止点的证据快照，以离线沙盒的形式保存，并要求模型预测截止点后的结果，从而在真实数据到来时实现可验证的评估，无需详尽的专家注释，同时分析人类与模型之间的不一致现象，针对诸如同行评审奖项等信号。此外，PoT还提供了一个受控的测试环境，用于评估基于代理的研究判断，比较使用工具的代理与非代理基线在提示消解和预算扩展下的表现。在跨越四个基准领域的30,000多个实例中，我们发现，与非代理基线相比，更高的交互预算通常会提高代理性能，而工具使用的好处则强烈依赖于任务。通过结合时间分区、未来可验证的目标与工具使用的离线沙盒，PoT支持对面向未来的科学思想判断任务的可扩展评估。
英文摘要：
Large language models are increasingly being used to assess and forecast research ideas, yet we lack scalable ways to evaluate the quality of models' judgments about these scientific ideas. Towards this goal, we introduce PoT, a semi-verifiable benchmarking framework that links scientific idea judgments to downstream signals that become observable later (e.g., citations and shifts in researchers' agendas). PoT freezes a pre-cutoff snapshot of evidence in an offline sandbox and asks models to forecast post-cutoff outcomes, enabling verifiable evaluation when ground truth arrives, scalable benchmarking without exhaustive expert annotation, and analysis of human-model misalignment against signals such as peer-review awards. In addition, PoT provides a controlled testbed for agent-based research judgments that evaluate scientific ideas, comparing tool-using agents to non-agent baselines under prompt ablations and budget scaling. Across 30,000+ instances spanning four benchmark domains, we find that, compared with non-agent baselines, higher interaction budgets generally improve agent performance, while the benefit of tool use is strongly task-dependent. By combining time-partitioned, future-verifiable targets with an offline sandbox for tool use, PoT supports scalable evaluation of agents on future-facing scientific idea judgment tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
