HindSight: 通过未来影响评估大语言模型生成的研究想法
HindSight: Evaluating LLM-Generated Research Ideas via Future Impact
AI总结：
本文提出HindSight框架，通过匹配生成想法与真实未来出版物，评估研究想法的质量，揭示LLM生成想法与实际研究影响的差异。
摘要：
评估AI生成的研究想法通常依赖于LLM判断者或人小组——两者都具有主观性且与实际研究影响脱节。我们引入HindSight，一种时间分割评估框架，通过将生成的想法与真实未来出版物匹配，并根据引用影响和会议接受度评分来衡量想法质量。使用时间截止点T，我们限制想法生成系统在T之前文献中进行生成，然后评估其输出与随后30个月内发表的论文相比的结果。在10个AI/ML研究主题上的实验揭示了显著的脱节：LLM作为判断者发现检索增强与常规想法生成之间无显著差异（p=0.584），而HindSight显示检索增强系统生成的想法得分高2.5倍（p<0.001）。此外，HindSight得分与LLM判断的创新性呈负相关（ρ=-0.29，p<0.01），表明LLM系统系统性地高估了那些从未在真实研究中实现的创新性想法。
英文摘要：
Evaluating AI-generated research ideas typically relies on LLM judges or human panels -- both subjective and disconnected from actual research impact. We introduce HindSight, a time-split evaluation framework that measures idea quality by matching generated ideas against real future publications and scoring them by citation impact and venue acceptance. Using a temporal cutoff~$T$, we restrict an idea generation system to pre-$T$ literature, then evaluate its outputs against papers published in the subsequent 30 months. Experiments across 10 AI/ML research topics reveal a striking disconnect: LLM-as-Judge finds no significant difference between retrieval-augmented and vanilla idea generation ($p{=}0.584$), while HindSight shows the retrieval-augmented system produces 2.5$\times$ higher-scoring ideas ($p{<}0.001$). Moreover, HindSight scores are \emph{negatively} correlated with LLM-judged novelty ($ρ{=}{-}0.29$, $p{<}0.01$), suggesting that LLMs systematically overvalue novel-sounding ideas that never materialize in real research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
