人类与语言模型如何推理创造力？一项比较分析
How do Humans and Language Models Reason About Creativity? A Comparative Analysis
AI总结：
该研究通过两项实验比较人类专家与大语言模型在创造力评分中的认知差异，发现无示例专家更依赖比较和罕见性，而大语言模型更侧重语义相似性，且示例会加剧其评分维度同质化。
摘要：
科学与工程领域的创造力评估日益依赖人类与人工智能的判断，但这些评价背后的认知过程与偏差仍未被充分理解。我们开展了两个实验，考察纳入带评分的示例解决方案如何影响创造力评价，采用一种细粒度标注协议，要求评分者解释其原创性评分，并对“远距性”（回答是否“远离”日常想法）、“罕见性”（回答是否稀有）和“巧妙性”等维度进行评分。在研究一中，我们分析了72位具有正式科学或工程训练背景的专家的创造力评分，比较了收到带评分示例解决方案的专家（示例组）与未收到示例的专家（无示例组）。计算文本分析显示，与有示例的专家相比，无示例专家使用了更多比较性语言（如“更好/更差”），并更强调解决方案的罕见性，表明他们可能更多依赖记忆检索来进行比较。在研究二中，对最先进的大语言模型的平行分析表明，模型在评定原创性时优先考虑想法的罕见性和远距性，提示其评价过程植根于想法的语义相似性。在示例条件下，尽管大语言模型预测真实原创性分数的准确率有所提高，但远距性、罕见性和巧妙性与原创性之间的相关性也大幅上升——最高达到0.99——表明大语言模型对各维度的评价出现了同质化。这些发现揭示了人类与人工智能如何推理创造力的重要启示，并表明不同群体在评分时所优先考虑的因素存在偏好分歧。
英文摘要：
Creativity assessment in science and engineering is increasingly based on both human and AI judgment, but the cognitive processes and biases behind these evaluations remain poorly understood. We conducted two experiments examining how including example solutions with ratings impact creativity evaluation, using a finegrained annotation protocol where raters were tasked with explaining their originality scores and rating for the facets of remoteness (whether the response is "far" from everyday ideas), uncommonness (whether the response is rare), and cleverness. In Study 1, we analyzed creativity ratings from 72 experts with formal science or engineering training, comparing those who received example solutions with ratings (example) to those who did not (no example). Computational text analysis revealed that, compared to experts with examples, no-example experts used more comparative language (e.g., "better/worse") and emphasized solution uncommonness, suggesting they may have relied more on memory retrieval for comparisons. In Study 2, parallel analyses with state-of-the-art LLMs revealed that models prioritized uncommonness and remoteness of ideas when rating originality, suggesting an evaluative process rooted around the semantic similarity of ideas. In the example condition, while LLM accuracy in predicting the true originality scores improved, the correlations of remoteness, uncommonness, and cleverness with originality also increased substantially -- to upwards of $0.99$ -- suggesting a homogenization in the LLMs evaluation of the individual facets. These findings highlight important implications for how humans and AI reason about creativity and suggest diverging preferences for what different populations prioritize when rating.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
