arXiv 2412.175962026-02-23cs.CLcs.AI

评估LLMs在科学想法生成中的发散思维能力：基于最小上下文

Evaluating LLMs' Divergent Thinking Capabilities for Scientific Idea Generation with Minimal Context

Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学人工智能学院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Bank of China（中国银行）；School of Engineering Science, University of Chinese Academy of Sciences（中国科学院大学工程科学学院）；State Key Laboratory of Nonlinear Mechanics, Institute of Mechanics, Chinese Academy of Sciences（中国科学院力学研究所非线性力学重点实验室）

Kai Ruan, Xuan Wang, Jixiang Hong, Peng Wang, Yang Liu, Hao Sun

备注：: Updated manuscript and title

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本研究提出LiveIdeaBench，通过单关键词提示评估LLMs在科学想法生成中的发散思维能力，揭示通用智能指标无法有效预测该能力，表明需专门的评估基准和不同的训练策略来提升科学想法生成能力。

摘要：

尽管大型语言模型（LLMs）在科学任务如文献分析和实验设计方面展现出显著能力（例如，准确提取论文中的关键发现或生成连贯的实验程序），现有的评估基准主要通过丰富的上下文输入来评估性能。我们引入了LiveIdeaBench，一个综合性的基准，通过使用单关键词提示来评估LLMs的科学想法生成能力，以评估发散思维能力。借鉴吉尔福德的创造力理论，我们的基准采用了一组最先进的LLMs动态面板，评估生成的想法在五个关键维度上的表现：原创性、可行性、流畅性、灵活性和清晰度。通过在涵盖22个科学领域的1180个关键词上对超过40个领先模型进行广泛实验，我们发现，由我们的基准测量的科学想法生成能力，无法由通用智能的标准指标预测。我们的结果表明，尽管在通用智能分数上存在显著差距，像QwQ-32B-preview这样的模型在创造性表现上可以与顶级模型如claude-3.7-sonnet:thinking相媲美。这些发现强调了对科学想法生成进行专门评估基准的必要性，并表明增强LLMs的想法生成能力可能需要不同于提高一般问题解决能力的训练策略，这可能使更多针对科学过程不同阶段的AI工具得以发展。

英文摘要：

While Large Language Models (LLMs) demonstrate remarkable capabilities in scientific tasks such as literature analysis and experimental design (e.g., accurately extracting key findings from papers or generating coherent experimental procedures), existing evaluation benchmarks primarily assess performance using rich contextual inputs. We introduce LiveIdeaBench, a comprehensive benchmark evaluating LLMs' scientific idea generation by assessing divergent thinking capabilities using single-keyword prompts. Drawing from Guilford's creativity theory, our benchmark employs a dynamic panel of state-of-the-art LLMs to assess generated ideas across five key dimensions: originality, feasibility, fluency, flexibility, and clarity. Through extensive experimentation with over 40 leading models across 1,180 keywords spanning 22 scientific domains, we reveal that the scientific idea generation capabilities measured by our benchmark, are poorly predicted by standard metrics of general intelligence. Our results demonstrate that models like QwQ-32B-preview achieve creative performance comparable to top-tier models such as claude-3.7-sonnet:thinking, despite significant gaps in their general intelligence scores. These findings highlight the need for specialized evaluation benchmarks for scientific idea generation and suggest that enhancing these idea generation capabilities in LLMs may require different training strategies than those used for improving general problem-solving abilities, potentially enabling a wider range of AI tools tailored for different stages of the scientific process.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。