arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2607.01233cs.CLcs.AI
收藏

衡量人类与LLM研究想法之间的差距

Measuring the Gap Between Human and LLM Research Ideas

Yale University(耶鲁大学);University of Chicago(芝加哥大学)

Ziyu Chen, Yilun Zhao, Arman Cohan

AI总结:

本文构建大规模评估框架,通过逆向工程提取论文核心想法,引入双轴研究品味分类法,发现LLM想法集中在桥梁式机会和综合方法,而人类想法分布更广,揭示两者系统性差距。

摘要:

LLM越来越多地被用于头脑风暴研究想法,但现有评估大多根据新颖性、可行性或专家偏好来评判单个想法。我们转而问:当前LLM生成的想法与人类研究人员之间有多大的差距?为了描述这一差距,我们为高质量人类研究论文的构思构建了一个大规模评估框架。对于每篇论文,我们逆向工程出一小组可能启发其核心思想的密切相关的先前工作。然后提示LLM从论文标题和摘要集合中生成一个新想法。我们引入一个双轴研究品味分类法,通过机会模式和研究范式来描绘每个想法的特征,并用它来量化人类与LLM想法之间的分歧。在不同LLM生成的想法集合中,我们观察到一致的分布差距:LLM想法不成比例地集中在桥梁式机会和综合方法上,而人类论文参考分布在构建差距和构建贡献的方式上更广泛。这一结果表明,强大的LLM可以产生一系列合理的想法,但该范围仍然比人类研究品味更窄,并且相对于人类研究品味存在系统性偏移。

英文摘要:

LLMs are increasingly used to brainstorm research ideas, but existing evaluations mostly judge individual ideas by novelty, feasibility, or expert preference. We instead ask: how far are current LLM-generated ideas from human researchers? To characterize this gap, we build a large-scale evaluation framework for ideation from high-quality human research papers. For each paper, we reverse-engineer a small set of closely related prior works that likely inspired its core idea. LLMs are then prompted to generate a new idea from the set of paper titles and summaries. We introduce a two-axis research-taste taxonomy to profile each idea by its opportunity pattern and research paradigm, and use it to quantify the divergence between human and LLM ideas. Across idea sets generated by different LLMs, we observe a consistent distributional gap: LLM ideas are disproportionately concentrated around bridge-like opportunities and synthesis methods, whereas the human paper reference distribution spreads more broadly across ways of framing gaps and constructing contributions. This result suggests that strong LLMs can produce a range of reasonable ideas, but that range remains narrower than, and systematically shifted relative to, human research taste.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑