大语言模型扩展已知研究但不创造新的研究方向
Large Language Models Extend Known Research But Do Not Create New Research Directions
AI总结:
该研究通过区分新颖性的距离与方向,发现LLM生成的假设虽与现有研究距离更远,但正交新颖性分数更低,仅擅长扩展研究前沿,无法创造新研究方向。
摘要:
自主研究系统的成功依赖于其生成的科学假设的质量。近期研究报告显示,人类评审者常将大语言模型(LLM)生成的创意评为比人类专家的创意更具新颖性,这让人们对AI加速科学发现抱有希望。但目前尚不清楚这些模型是否能做的不止于扩展现有知识:LLM能否通过揭示新的研究方向推动真正的创新,还是仅限于重复和重组科学已有的内容?这一未解决的问题是关于AI与科学未来讨论的核心。为解决该问题,我们区分了两种新颖性类型:距离(衡量假设与现有研究的差距)和方向(捕捉假设是否追求超出已有研究的新探究方向),这两种衡量方式在LLM生成的成果中存在显著差异。我们使用预训练的科学论文嵌入模型将科学论文表示在共享概念空间中,在该空间内嵌入了六个自然科学领域1325个研究主题的2690万篇人类撰写论文,以及八个模型生成的524308个LLM假设。距离使用现有指标衡量;为量化方向,我们引入了正交新颖性分数(ONS)。LLM生成的创意与现有研究的距离始终比人类撰写论文更远,当为LLM提供距离主题中心最远的论文时,距离最多可远16%;但LLM假设的中位ONS比同一主题的已发表论文低15%-23%。因此,LLM能在已知内容上走得更远,但仍处于已确立的方向内,该模式在各种配置变化下均持续存在。结果表明,LLM擅长扩展前沿领域,但新方向的创造仍由人类科学家主导。
英文摘要:
The success of autonomous research systems relies on the quality of the scientific hypotheses they generate. Recent studies report that human reviewers often rate LLM-generated ideas as more novel than those from human experts, raising hope that AI can accelerate scientific discovery. Yet it is unclear whether these models can do more than extend existing knowledge. Can LLMs drive genuine innovation by revealing new research directions, or are they limited to repeating and recombining what science already knows? This unresolved question lies at the heart of discussions about AI and the future of science. To address this, we distinguish two types of novelty: distance, which measures how far a hypothesis lies from existing work, and direction, which captures whether the hypothesis pursues new lines of inquiry beyond those already present. These two measures diverge sharply for LLM-generated work. We represent scientific papers in a shared conceptual space using a pretrained scientific-paper embedding model. Within this space, we embed 26.9 million human-authored papers across 1,325 research topics in six natural-science fields and 524,308 LLM-generated hypotheses from eight models. Distance is measured using existing metrics. To quantify direction, we introduce the orthogonal novelty score (ONS). LLMs consistently generate ideas more distant from existing work than human-authored papers, up to 16\% further when the LLM is given the papers farthest from the centre of the topic. However, LLM hypotheses have a 15--23\% lower median ONS than published papers from the same topic. Thus, LLMs move further from what is already known, but remain within established directions. This pattern persists across changes in various configurations. As a result, LLMs excel at extending the frontier but the creation of new directions is still led by human scientists.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。