创造性偏好优化
Creative Preference Optimization
AI总结：
本文提出创造性偏好优化（CrPO）方法，通过模块化地将多维度创造力信号注入偏好优化目标，并利用大规模人类偏好数据集MuCE训练模型，在保持输出质量的同时显著提升LLM生成内容的新颖性、多样性和惊喜感，超越GPT-4o等强基线。
摘要：
尽管大型语言模型（LLMs）在自然语言生成任务中表现出色，但其生成真正创造性内容的能力——以新颖性、多样性、惊喜感和质量为特征——仍然有限。现有增强LLM创造力的方法往往只关注多样性或特定任务，未能以可推广的方式解决创造力的多面性。在本工作中，我们提出创造性偏好优化（Creative Preference Optimization，CrPO），这是一种新颖的对齐方法，以模块化方式将多个创造力维度的信号注入偏好优化目标中。我们使用CrPO和MuCE训练并评估了多个模型的创造力增强版本，MuCE是一个新的大规模人类偏好数据集，涵盖超过200,000条人类生成的回答和来自30多项心理创造力评估的评分。我们的模型在自动评估和人类评估中均超越了包括GPT-4o在内的强基线，生成了更新颖、更多样、更令人惊喜的内容，同时保持高输出质量。在NoveltyBench上的额外评估进一步证实了我们方法的可推广性。综上，我们的结果表明，在偏好框架内直接优化创造力是提升LLM创造能力且不牺牲输出质量的一个有前景的方向。
英文摘要：
While Large Language Models (LLMs) have demonstrated impressive performance across natural language generation tasks, their ability to generate truly creative content-characterized by novelty, diversity, surprise, and quality-remains limited. Existing methods for enhancing LLM creativity often focus narrowly on diversity or specific tasks, failing to address creativity's multifaceted nature in a generalizable way. In this work, we propose Creative Preference Optimization (CrPO), a novel alignment method that injects signals from multiple creativity dimensions into the preference optimization objective in a modular fashion. We train and evaluate creativity-augmented versions of several models using CrPO and MuCE, a new large-scale human preference dataset spanning over 200,000 human-generated responses and ratings from more than 30 psychological creativity assessments. Our models outperform strong baselines, including GPT-4o, on both automated and human evaluations, producing more novel, diverse, and surprising generations while maintaining high output quality. Additional evaluations on NoveltyBench further confirm the generalizability of our approach. Together, our results demonstrate that directly optimizing for creativity within preference frameworks is a promising direction for advancing the creative capabilities of LLMs without compromising output quality.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
