arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2607.01152cs.CL
收藏

AGC-Bench:衡量人工通用创造力

AGC-Bench: Measuring Artificial General Creativity

Pennsylvania State University(宾夕法尼亚州立大学);University of Massachusetts Lowell(马萨诸塞大学洛厄尔分校);University of Amsterdam(阿姆斯特丹大学);Amazon AGI(亚马逊AGI);Dartmouth College(达特茅斯学院)
Roger Beaty, Vijeta Deshpande, Clin K. Y. Lai, Anna Attuch, Namrata Shivagunde, Swastik Roy, Rajkumar Pujari, Paul V. DiStefano, Sherin Muckatira, Claire E. Ste… 展开作者

Roger Beaty, Vijeta Deshpande, Clin K. Y. Lai, Anna Attuch, Namrata Shivagunde, Swastik Roy, Rajkumar Pujari, Paul V. DiStefano, Sherin Muckatira, Claire E. Stevenson, Mikhail Gronas, Anna Rumshisky

AI总结:

提出AGC-Bench基准,通过系统文献综述和标准化框架评估LLM创造力,发现单一创造力因子'c',并验证提示'要有创造力'比推理更有效。

摘要:

创造力研究一直争论创造力是领域特定的(例如,视觉、写作、科学),还是可以从心理测量学上与一般智力分离。这两个问题现在都适用于LLM,但统一的AI创造力基准仍然难以捉摸。我们引入了AGC-Bench,一个基于AI创造力文献系统综述(筛选了3101篇论文,识别了497个基准)构建的人工通用创造力基准,并配有一个代理框架,将特异的代码库转换为HELM标准化的基准。首次发布涵盖了78个数据集,涵盖头脑风暴、问题解决、STEM、叙事、比喻语言和幽默。为了解决LLM作为评判者的偏见,我们应用了评判者响应理论——对评判者宽严程度的心理测量校准;然后我们在三个前沿LLM的偏差校正评分上微调Qwen3-30B,产生AGC-Judge,一个开放权重的模型,能够稳健地评分它未训练过的新创造力基准。结果显示,前沿模型位于AGC-Bench排行榜顶部,开放模型紧随其后。LLM表现出不同的创造力优势,在某些领域(例如写作)排名高于其他领域(例如科学构思)。大量实验得出三个主要发现。首先,对83个LLM进行因子分析,我们恢复了一个单一的创造力因子'c',类似于一般智力的'g'因子,解释了81.5%的方差,与一般知识/推理相关但可分离。其次,我们表明提示模型“要有创造力”比启用推理更能提升其性能,证明该基准追踪的是创造力而非一般能力。第三,在人类匹配的子集上,我们发现顶级人类在创造力上仍然领先于顶级LLM。我们发布了AGC-Bench,附带公共排行榜、AGC-Judge和人类数据,作为大规模衡量AI创造力的开放基础设施。

英文摘要:

Creativity research has debated whether creativity is domain-specific (e.g., visual, writing, science), and if it is psychometrically separable from general intelligence. Both questions now apply to LLMs, but a unified benchmark of AI creativity remains elusive. We introduce AGC-Bench, an artificial general creativity benchmark built from a systematic review of the AI creativity literature (3,101 papers screened, 497 benchmarks identified), paired with an agentic harness that converts idiosyncratic codebases into HELM-standardized benchmarks. The first release covers 78 datasets spanning brainstorming, problem solving, STEM, narrative, figurative language, and humor. To address bias in LLM-as-judge, we apply Judge Response Theory -- a psychometric calibration of judge leniency/severity; we then fine-tune Qwen3-30B on the bias-corrected ratings of three frontier LLMs to produce AGC-Judge, an open-weight model that robustly scores new creativity benchmarks it was not trained on. Results reveal frontier models at the top of the AGC-Bench leaderboard, with open models close behind. LLMs show different creative strengths, ranking higher on some domains (e.g., writing) than others (e.g., scientific ideation). Extensive experiments yield three main findings. First, applying factor analysis across 83 LLMs, we recover a single creativity factor 'c', analogous to the 'g' factor of general intelligence, that explains 81.5% of variance, related to but separable from general knowledge/reasoning. Second, we show that prompting models to "be creative" boosts their performance far more than enabling reasoning, evidence that the benchmark tracks creativity over general ability. Third, on a human-matched subset, we find the top human still leads the top LLM on creativity. We release AGC-Bench with a public leaderboard, AGC-Judge, and human data as open infrastructure for measuring AI creativity at scale.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑