组合创造力：一般化能力的新前沿
Measuring Combinatorial Creativity in Large Language Models and Humans
AI总结：
本文提出组合创造力的概念，通过理论框架和算法任务评估LLMs的创造力，并发现最优模型参数和新颖性-实用性权衡对提升创造力的重要性。
摘要：
人工智能系统，特别是大型语言模型（LLMs），越来越多地被用于创造性任务，如科学想法生成，这构成了从训练数据中进行一般化的一种形式，而现有概念框架未涉及。尽管其与组合性一般化（CG）有相似之处，但组合创造力（CC）是一种开放性能力。与评估准确性或正确性以固定目标相矛盾，我们提出了一种理论框架和算法任务，以评估输出的新颖性和实用性程度。从这里，我们做出了几个重要的实证贡献：（1）我们获得了对LLMs创造力缩放行为的首次见解。（2）我们发现，在固定计算预算下，存在最优的模型深度和宽度以增强创造力。（3）我们发现，想法生成与执行之间的差距，即LLMs在生成新颖的科学想法方面表现优异，但在确保其实际可行性方面存在困难，可能由创造力算法更基本的新颖性-实用性权衡特征所解释。尽管我们的发现持续到100M规模，但前沿模型如今已进入数十亿参数级别。因此，我们的概念框架和实证发现最好作为理解并改进当前前沿模型创造力的起点，因为我们开始弥合人与机器智能之间的差距。
英文摘要：
Combinatorial creativity, the process of combining familiar ideas in unfamiliar ways, is one of the foremost engines of scientific, technological, and artistic progress. This work introduces Kombine, an LLM benchmark for combinatorial creativity tasking models to discover surprising, original, and useful associations, analogies, and blends between unrelated entities, and to use those analogies and blends to invent novel concepts by re-combining the parts of existing ones. In a large-scale study across 35 LLMs, we find that: (1) LLMs are 10-16 percentage points better at discovering associations and analogies than blends, and (2) exhibit an abstraction bottleneck on blending, in which even frontier LLMs struggle to find a shared abstraction between inputs and instantiate a coherent blended space. (3) LLMs also exhibit inventive homogeneity, in which one in five concepts invented for a given item reuses properties invented by another model. Moreover, (4) models are $14\times$ more likely to reuse other invented properties on blends than analogies, and (5) models from the same provider are $2.3\times$ more likely to reuse properties than those from different providers. Next, in a comparative study involving 75 humans on the same tasks, we find that (6) humans score below nearly every LLM on each task and (7) both humans and LLMs find blending most difficult, but association is easier than analogy for LLMs, while analogy is easier than association for humans. Lastly, (8) humans and LLMs access distinct regions of semantic space on each task, suggesting that combinatorial creativity could benefit from systematic human-AI collaboration to enhance conceptual diversity. In summary, we provide a framework to assess progress towards LLMs, AI systems, and human-AI collaborations more broadly, that can elicit combinatorial creativity in a setting relevant to scientific and technological innovation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
