LLM Jaggedness Unlocks Scientific Creativity
LLM Jaggedness Unlocks Scientific Creativity
AI总结：
本文研究了大型语言模型（LLMs）在科学创意生成中的不规则进步现象，提出了SciAidanBench基准测试集，通过评估不同模型在科学问题上的创意生成能力，揭示了模型在跨任务、提示和领域层面的不均衡表现，并展示了如何通过推理计算、知识聚合和头脑风暴等机制利用这种不规则性来提升科学创造力。
摘要：
随着人工智能的发展，模型的改进并非均匀发生，而是呈现出不规则的进展方式，其能力在不同任务、领域和模型规模上增长不均。在本文中，我们通过科学创意生成的视角来研究这种动态的不规则性。我们引入了SciAidanBench，这是一个设计用于衡量大型语言模型（LLMs）科学创造力的开放性科学问题基准测试集。给定一个科学问题，模型被要求生成尽可能多的唯一且连贯的想法，总的有效响应数量作为创意潜力的代理。评估19个基础模型（共30个变体，包括推理版本）后，我们发现不规则性既体现在模型之间，也体现在模型内部。首先，在通用创造力与科学创造力的跨任务比较中，通用创造力的提升并不均匀地转化为科学创造力，揭示了模型在不同任务上的能力差异。其次，在提示层面，更强的模型并不均匀地提升，而是表现出高度的变异性，某些问题上出现创意爆发，而其他问题上表现有限。第三，在领域层面，单个模型在不同科学子领域中的表现不均衡，反映了内部能力的碎片化。最后，我们展示了这种不规则性可以被利用。我们探索了推理时间计算、知识聚合和头脑风暴等机制，以有效结合模型并构建超越单个模型的元模型集合。我们的结果表明，不规则性不应被视为限制，而是一种资源，是AI进步的结构性特征，当被理解和利用时，可以放大LLM驱动的科学创造力。
英文摘要：
As artificial intelligence advances, models are not improving uniformly. Instead, progress unfolds in a jagged fashion, with capabilities growing unevenly across tasks, domains, and model scales. In this work, we examine this dynamic jaggedness through the lens of scientific idea generation. We introduce SciAidanBench, a benchmark of open-ended scientific questions designed to measure the scientific creativity of large language models (LLMs). Given a scientific question, models are asked to generate as many unique and coherent ideas as possible, with the total number of valid responses serving as a proxy for creative potential. Evaluating 19 base models across 8 providers (30 total variants including reasoning versions), we find that jaggedness manifests both across models and within models. First, in a cross-task comparison between general and scientific creativity, improvements in general creativity do not translate uniformly to scientific creativity, revealing divergent capability profiles across models. Second, at the prompt level, stronger models do not improve uniformly; instead, they exhibit high variability, with bursts of creativity on some questions and limited performance on others. Third, at the domain level, individual models display uneven strengths across scientific subfields, reflecting fragmented internal capability profiles. Finally, we show that this jaggedness can be harnessed. We explore mechanisms of inference-time compute, knowledge pooling, and brainstorming to combine models effectively and construct meta-model ensembles that outperform any single model. Our results position jaggedness not as a limitation, but as a resource, a structural feature of AI progress that, when understood and leveraged, can amplify LLM-driven scientific creativity.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
