评估人类与大型语言模型的创意短篇故事生成能力
Evaluating Creative Short Story Generation in Humans and Large Language Models
AI总结：
本文通过五句提示词故事任务，对60个LLM与60个人类进行多维度自动评估和人机评分，发现LLM语言风格复杂但新颖性、意外性和多样性不及人类，且非专家与LLM评分存在偏差。
摘要：
故事创作是人类想象力的一个基本方面，它在很大程度上依赖创造力来产生新颖、有效且令人惊讶的叙事。尽管大型语言模型（LLM）已经展现出生成高质量故事的能力，但其创意故事写作能力仍未得到充分探索。在本研究中，我们使用一项基于提示词的五句创意故事写作任务，对60个LLM和60个人类的短篇故事生成创造力进行了系统分析。我们采用多项指标，从新颖性、意外性、多样性和语言复杂性等多个创造力维度，自动评估模型与人类生成的故事。我们还收集了非专家和专家人类评分者以及LLM给出的创造力评分和图灵测试分类。自动化指标显示，LLM能够生成风格复杂的故事，但与普通人类作者相比，其在新颖性、意外性和多样性方面往往存在不足。专家评分总体上与自动化指标一致。然而，LLM和非专家却认为LLM生成的故事比人类生成的故事更具创造力。我们讨论了这些评分差异产生的原因与方式，以及它们对人类和人工创造力的影响。
英文摘要：
Story-writing is a fundamental aspect of human imagination, relying heavily on creativity to produce narratives that are novel, effective, and surprising. While large language models (LLMs) have demonstrated the ability to generate high-quality stories, their creative story-writing capabilities remain under-explored. In this work, we conduct a systematic analysis of creativity in short story generation across 60 LLMs and 60 people using a five-sentence cue-word-based creative story-writing task. We use measures to automatically evaluate model- and human-generated stories across several dimensions of creativity, including novelty, surprise, diversity, and linguistic complexity. We also collect creativity ratings and Turing Test classifications from non-expert and expert human raters and LLMs. Automated metrics show that LLMs generate stylistically complex stories, but tend to fall short in terms of novelty, surprise and diversity when compared to average human writers. Expert ratings generally coincide with automated metrics. However, LLMs and non-experts rate LLM stories to be more creative than human-generated stories. We discuss why and how these differences in ratings occur, and their implications for both human and artificial creativity.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
