CS4：通过控制故事写作约束数量自动衡量大语言模型的创造力
CS4: Measuring the Creativity of Large Language Models Automatically by Controlling the Number of Story-Writing Constraints
AI总结：
本研究提出新型基准CS4，通过调整故事写作提示的约束数量控制明确度，无需人工标注即可间接衡量大语言模型的创造力，实验揭示了不同模型的创造力表现差异及人类反馈学习的作用局限。
摘要：
评估大语言模型（LLM）在故事写作中的创造力颇具难度，因为LLM生成的故事可能看似富有创意，却与其庞大专有训练语料库中的某些现有故事高度相似。为攻克这一难题，我们推出了一套具备不同提示明确度层级的新型基准数据集：CS4（通过控制合成约束明确度比较故事创作技能，Comparing the Skill of Creating Stories by Controlling the Synthesized Constraint Specificity）。通过增加提示中的要求/约束数量，我们能够提升提示的明确度，阻碍LLM复述其训练数据中的优质叙事内容。如此一来，CS4无需人工标注即可让我们间接衡量LLM的创造力。\n我们在LLaMA、Gemma和Mistral上开展的实验，不仅凸显了LLM在处理高度明确的提示时面临的创造力挑战，还揭示出不同LLM在不同约束数量下的表现差异极大，且在模型的指令遵循能力与叙事连贯性之间实现了不同的平衡。此外，我们在OLMo上的实验表明，人类反馈学习（LHF）能够帮助LLM从训练数据中筛选出更优质的故事，但在提升LLM生成训练语料中未出现的创意故事的能力方面作用有限。该基准已在https://github.com/anirudhlakkaraju/cs4_benchmark 发布。
英文摘要：
Evaluating the creativity of large language models (LLMs) in story writing is difficult because LLM-generated stories could seemingly look creative but be very similar to some existing stories in their huge and proprietary training corpus. To overcome this challenge, we introduce a novel benchmark dataset with varying levels of prompt specificity: CS4 ($\mathbf{C}$omparing the $\mathbf{S}$kill of $\mathbf{C}$reating $\mathbf{S}$tories by $\mathbf{C}$ontrolling the $\mathbf{S}$ynthesized $\mathbf{C}$onstraint $\mathbf{S}$pecificity). By increasing the number of requirements/constraints in the prompt, we can increase the prompt specificity and hinder LLMs from retelling high-quality narratives in their training data. Consequently, CS4 empowers us to indirectly measure the LLMs' creativity without human annotations. Our experiments on LLaMA, Gemma, and Mistral not only highlight the creativity challenges LLMs face when dealing with highly specific prompts but also reveal that different LLMs perform very differently under different numbers of constraints and achieve different balances between the model's instruction-following ability and narrative coherence. Additionally, our experiments on OLMo suggest that Learning from Human Feedback (LHF) can help LLMs select better stories from their training data but has limited influence in boosting LLMs' ability to produce creative stories that are unseen in the training corpora. The benchmark is released at https://github.com/anirudhlakkaraju/cs4_benchmark.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
