arXivDaily arXiv每日学术速递 周一至周五更新
​通知:联邦节假日调休:哥伦布日 / 原住民日(Columbus Day / Indigenous Peoples' Day),今日(13日)无更新。
arXiv 2509.09702cs.CLcs.AIcs.HC

Creativity Benchmark:面向大型语言模型的营销创意基准

Creativity Benchmark: A benchmark for marketing creativity for large language models

Ninad Bhat, Kieran Browne, Pip Bingemann

AI总结:

该研究提出覆盖100个品牌、三类提示的营销创意基准 Creativity Benchmark,通过678名创意人员的上万次成对比较表明LLM差距很小,且LLM评判者不可靠。

摘要:

我们推出 Creativity Benchmark,这是一个针对大型语言模型(LLM)在营销创意方面表现的评估框架。该基准涵盖100个品牌(12个类别)和三种提示类型(Insights、Ideas、Wild Ideas)。来自678名在职创意人员的人类成对偏好包含11,012项匿名比较,并使用 Bradley-Terry 模型进行分析;结果显示各模型表现紧密聚集,没有模型在不同品牌或提示类型中占据主导:最高与最低评分之间的差距为 $Δθ\approx 0.45$,这意味着正面交锋胜率为 $0.61$;评分最高的模型仅有约61%的时间击败评分最低的模型。我们还使用余弦距离分析模型多样性,以捕捉模型内和模型间的变化,以及对提示重构的敏感性。将三种 LLM-as-judge 设置与人类排名进行比较后发现,二者相关性较弱且不一致,并存在因评判者而异的偏差,这凸显自动评判无法替代人类评估。传统创造力测试也只能部分迁移到受品牌约束的任务。总体而言,结果强调了专家人类评估和具备多样性意识的工作流程的必要性。

英文摘要:

We introduce Creativity Benchmark, an evaluation framework for large language models (LLMs) in marketing creativity. The benchmark covers 100 brands (12 categories) and three prompt types (Insights, Ideas, Wild Ideas). Human pairwise preferences from 678 practising creatives over 11,012 anonymised comparisons, analysed with Bradley-Terry models, show tightly clustered performance with no model dominating across brands or prompt types: the top-bottom spread is $Δθ\approx 0.45$, which implies a head-to-head win probability of $0.61$; the highest-rated model beats the lowest only about $61\%$ of the time. We also analyse model diversity using cosine distances to capture intra- and inter-model variation and sensitivity to prompt reframing. Comparing three LLM-as-judge setups with human rankings reveals weak, inconsistent correlations and judge-specific biases, underscoring that automated judges cannot substitute for human evaluation. Conventional creativity tests also transfer only partially to brand-constrained tasks. Overall, the results highlight the need for expert human evaluation and diversity-aware workflows.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑