在不同领域评估文本创造力：一个数据集和大语言模型评估器
Evaluating Text Creativity across Diverse Domains: A Dataset and Large Language Model Evaluator
AI总结：
本文提出了一种基于CreataSet数据集的CrEval评估器，通过结合人类和合成数据提升大语言模型的创造力评估效果。
摘要：
评估创造力仍然是大语言模型（LLMs）面临的挑战性前沿。当前的评估严重依赖于低效且昂贵的人类判断，阻碍了提高机器创造力的进步。虽然存在自动化方法，从心理测试到启发式或提示基于的方法，但它们通常缺乏通用性或与人类判断的一致性。为了解决这些问题，我们提出了一种新的成对比较框架，用于评估文本创造力，该框架利用共享的上下文指令来提高评估的一致性。我们引入了CreataSet，一个大规模的数据集，包含10万多个水平人类和100万多个合成的创造性指令-响应对，涵盖多样化的开放领域任务。通过在CreataSet上训练，我们开发了一个基于LLM的评估器，名为CrEval。CrEval在与人类判断的一致性方面显示出显著优于现有方法。实验结果强调了整合人类和合成数据以训练高度稳健的评估器的重要性，并展示了CrEval在提升LLMs创造力方面的实际用途。
英文摘要：
Creativity evaluation remains a challenging frontier for large language models (LLMs). Current evaluations heavily rely on inefficient and costly human judgments, hindering progress in enhancing machine creativity. While automated methods exist, ranging from psychological testing to heuristic- or prompting-based approaches, they often lack generalizability or alignment with human judgment. To address these issues, we propose a novel pairwise-comparison framework for assessing textual creativity that leverages shared contextual instructions to improve evaluation consistency. We introduce CreataSet, a large-scale dataset with 100K+ human-level and 1M+ synthetic creative instruction-response pairs spanning diverse open-domain tasks. Through training on CreataSet, we develop an LLM-based evaluator named CrEval. CrEval demonstrates remarkable superiority over existing methods in alignment with human judgments. Experimental results underscore the indispensable significance of integrating both human and synthetic data to train highly robust evaluators, and showcase the practical utility of CrEval in boosting the creativity of LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
