CREATE：测试大型语言模型的联想创造力
CREATE: Testing LLMs for Associative Creativity
AI总结：
CREATE基准测试评估模型的创造性联想能力，要求生成高特异性和多样性的概念连接路径，展现模型在复杂搜索空间中的创造力表现。
摘要：
创造性的一个关键组成部分是联想推理：即能够绘制出新颖 yet 有意义的概念之间的连接。我们介绍了CREATE，一个用于评估模型创造性联想能力的基准测试。CREATE要求模型生成连接概念的路径集。路径应具有高特异性（概念连接的区分度和接近性）和高多样性（与其他路径的差异性），并且模型如果能生成更多强而多样化的路径则得分更高。此任务与现实中的创造性任务如假设生成具有相似的需求，包括极其庞大的搜索空间，但能够收集到一个可观的基准测试集，并具有客观的答案评分。对前沿模型的评估显示，最强的模型在创造性实用性上优于其他模型，由于答案的高多样性以及搜索的复杂性，使基准测试饱和难以实现。此外，我们的结果表明，即使在高token预算下，思考模型也不总是比其他模型在本任务上更有效。最近的创造性提示方法提供了一些但有限的额外改进。CREATE为开发新方法以提高模型的联想创造力能力提供了一个沙盒环境。
英文摘要：
A key component of creativity is associative reasoning: the ability to draw novel yet meaningful connections between concepts. We introduce CREATE, a benchmark designed to evaluate models' capacity for creative associative reasoning. CREATE requires models to generate sets of paths connecting concepts in a model's parametric knowledge. Paths should have high specificity (distinctiveness and closeness of the concept connection) and high diversity (dissimilarity from other paths), and models are scored more highly if they produce a larger set of strong, diverse paths. This task shares demands of real creativity tasks like hypothesis generation, including an extremely large search space, but enables collection of a sizable benchmark with objective answer grading. Evaluation of frontier models shows that the strongest models achieve higher creative utility than others, with the high multiplicity of answers and complexity of the search making benchmark saturation difficult to achieve. Furthermore, our results illustrate that thinking models are not always more effective on our task, even with high token budgets. Recent approaches for creative prompting give some but limited additional improvement. CREATE provides a sandbox for developing new methods to improve models' capacity for associative creativity.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
