重新思考创造力评估：对现有创造力评估方法的批判性分析
Rethinking Creativity Evaluation: A Critical Analysis of Existing Creativity Evaluations
AI总结：
本文批判性分析现有创造力评估方法，发现其在不同领域和度量标准间存在局限性，需发展更稳健的评估框架。
摘要：
我们考察、分析并比较了四种代表性的创造力度量标准——困惑度、LLM作为评判者、创造力指数（CI；测量n-gram与网络语料库的重叠度）以及语法模板（检测常见词性模式的重复）——在多样化的创造性领域，如创造性写作、非传统问题解决和研究构想中的表现。对于每个领域，我们编纂了包含人类对创造性与非创造性示例的对齐数据集，并评估每种度量标准在区分两组数据集的能力。我们的分析揭示了在不同领域和度量标准之间存在有限的一致性，因为能够在某一领域区分创造力的度量标准在其他领域却失败（例如，CI在创造性写作中正确区分，但在问题解决中却失败），且不同的度量标准往往在相同的数据点上产生分歧（例如，CI认为一组更具有创造性，而困惑度却认为另一组更具有创造性）。我们强调了关键的局限性，例如困惑度反映的是流畅性而非新颖性；LLM作为评判者在微小提示变化下产生不一致的判断，并倾向于特定标签；CI主要测量词汇多样性，对实现选择高度敏感；而语法模板在以公式化语言为主设置中效果不佳。我们的发现强调了需要更稳健、更具普遍性的评估框架，以更好地与人类对创造力的判断相一致。
英文摘要：
We examine, analyze, and compare four representative creativity measures--perplexity, LLM-as-a-Judge, the Creativity Index (CI; measuring n-gram overlap with web corpora), and syntactic templates (detecting repetition of common part-of-speech patterns)--across the diverse creative domains, such as creative writing, unconventional problem-solving, and research ideation. For each domain, we compile datasets with human-aligned creative and uncreative examples and evaluate each metric's ability to discriminate between the two sets. Our analyses reveal limited consistency both across domains and metrics, as metrics that distinguish creativity in one domain fail in others (e.g., CI correctly distinguishes in creative writing but fails in problem-solving), and different metrics often disagree on the same data points (e.g., CI suggests one set to be more creative, while perplexity indicates the other set to be more creative.) We highlight key limitations, such as perplexity reflecting fluency rather than novelty; LLM-as-a-Judge producing inconsistent judgments under minor prompt variations and exhibiting bias towards particular labels; CI primarily measuring lexical diversity, with high sensitivity to implementation choices; and syntactic templates being ineffective in settings dominated by formulaic language. Our findings underscore the need for more robust, generalizable evaluation frameworks that better align with human judgments of creativity.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
