大型语言模型能否预测研究人员接下来会研究什么？
Can Large Language Models Forecast What Researchers Study Next?
浏览论文内容
中文总结 AI 辅助
该研究推出 IdeaForecastBench 基准，对比不同大模型的思路预测性能，发现 Qwen2.5 表现优于 GPT-4.1，为研究领域后续研究方向预测提供统一任务。
中文摘要 AI 辅助
大型语言模型越来越多地生成研究思路，但在生成时判断其新颖性或可行性，并不能确定它们是否能预测后续工作。我们推出 IdeaForecastBench 用于评估研究思路预测：给定某个领域截至某一截止时间的文献，系统生成最多五个排名的思路，将其与后续论文进行对比评估。该基准包含 52 个主题的 624 个滚动回合，采用固定的“先检索后判断”协议，并分别报告两名评判者的结果。我们在 GPT-4.1、Qwen2.5-7B/14B 和 Qwen3.5-9B 以及学习型模式分解预测器（Mode-Decomposition Forecaster, MDF）上对比了五种历史压缩策略。在主要评判者 GPT-4.1-mini 下，Summary 在所有四个主干模型上的 Hit@5 和 Precision@5 均优于 Direct；Qwen2.5 的得分高于 GPT-4.1，而 Qwen3.5 的得分低于 GPT-4.1。一项不依赖结果的评估发现，Qwen2.5 生成的预测范围更广，但未明确该广度对其优势的贡献程度。阈值和评判者诊断进一步阐明了将实现视为精确预测的局限性。IdeaForecastBench 为研究某个领域后续会追求哪些研究思路以及该结果的可测量性提供了统一任务。
英文摘要
Large language models increasingly generate research ideas, yet judging their novelty or feasibility at generation time does not establish whether they anticipate subsequent work. We introduce IdeaForecastBench to evaluate research idea forecasting. Given a community's literature up to a cutoff, a system produces up to five ranked ideas, which are evaluated against later papers. The benchmark comprises 624 rolling episodes across 52 topics, with a fixed retrieve-then-judge protocol and separately reported results from two judges. We compare five history-compression strategies across GPT-4.1, Qwen2.5-7B/14B, and Qwen3.5-9B, together with a learned Mode-Decomposition Forecaster (MDF). Under the primary GPT-4.1-mini judge, Summary improves on Direct in Hit@5 and Precision@5 across all four backbones. Qwen2.5 scores above GPT-4.1, whereas Qwen3.5 scores below it. An outcome-blind assessment finds that Qwen2.5 produces broader forecasts, but does not identify how much breadth contributes to its advantage. Threshold and judge diagnostics further clarify the limits of interpreting realization as precise anticipation. IdeaForecastBench provides a common task for studying which research ideas a community subsequently pursues and how reliably this outcome can be measured.
发表机构
- University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
机构由 AI 辅助整理，请以论文原文为准。
