大语言模型（LLMs）在替代用途创造性评估上是否存在一致性？
Do LLMs Agree on the Creativity Evaluation of Alternative Uses?
AI总结：
本文研究大语言模型（LLMs）在替代用途测试（AUT）回答的创造性评估上的一致性，通过基准数据集实验四种LLMs，发现模型间高度一致且不偏向自身回答，验证了其在自动化创造性评估中的可靠性与公正性。
摘要：
本文研究了大语言模型（LLMs）在评估替代用途测试（AUT）回答的创造性时是否表现出一致性。尽管LLMs越来越多地用于评估创造性内容，但先前的研究主要集中在单一模型评估由同一模型或人类生成的回答。本文探讨了LLMs是否能公正、准确地评估自身及其他模型生成输出的创造性。我们使用一个按创造性水平（普通、创造性和高度创造性）分类的AUT回答的基准数据集，对四种最先进的LLMs评估这些输出的情况进行了实验。我们测试了评分和排名两种方法，并采用综合和分段两种评估设置，以检验LLMs在替代用途创造性评估上是否存在一致性。结果显示模型间具有高度一致性，模型间的Spearman相关系数平均超过0.7，与基准的相关系数超过0.77，表明LLMs在替代用途创造性评估上具有高度一致性并验证了其可靠性。值得注意的是，模型不偏向自身的回答，而是对其他模型生成的替代用途给出相似的创造性评估分数或排名。这些发现表明LLMs在创造性评估中表现出公正性和高度一致性，为其在自动化创造性评估中的应用提供了良好的启示。
英文摘要：
This paper investigates whether large language models (LLMs) show agreement in assessing creativity in responses to the Alternative Uses Test (AUT). While LLMs are increasingly used to evaluate creative content, previous studies have primarily focused on a single model assessing responses generated by the same model or humans. This paper explores whether LLMs can impartially and accurately evaluate creativity in outputs generated by both themselves and other models. Using an oracle benchmark set of AUT responses, categorized by creativity level (common, creative, and highly creative), we experiment with four state-of-the-art LLMs evaluating these outputs. We test both scoring and ranking methods and employ two evaluation settings (comprehensive and segmented) to examine if LLMs agree on the creativity evaluation of alternative uses. Results reveal high inter-model agreement, with Spearman correlations averaging above 0.7 across models and reaching over 0.77 with respect to the oracle, indicating a high level of agreement and validating the reliability of LLMs in creativity assessment of alternative uses. Notably, models do not favour their own responses, instead they provide similar creativity assessment scores or rankings for alternative uses generated by other models. These findings suggest that LLMs exhibit impartiality and high alignment in creativity evaluation, offering promising implications for their use in automated creativity assessment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
