AI中的回声：量化大语言模型输出的情节多样性缺失
Echoes in AI: Quantifying lack of plot diversity in LLM outputs
AI总结：
本研究针对大语言模型故事生成的情节多样性不足问题，提出Sui Generis自动评分指标量化情节独特性，验证了LLM生成内容存在跨模型、跨批次的情节重复现象，且该指标与人类惊喜度判断呈中等相关。
摘要：
随着大语言模型（LLM）的快速发展，LLM在创意内容构思与生成中的应用日益广泛。一个关键问题随之浮现：当前的LLM能否提供足够多样的创意，真正助力集体创造力？我们针对故事生成任务，检验了两款最先进的LLM——GPT-4与LLaMA-3，发现LLM生成的故事往往包含在多次生成中重复出现的情节元素。为量化这一现象，我们提出了Sui Generis分数（独特性分数），这是一种自动指标，用于衡量在同一提示词下、由同一LLM生成的不同故事线中，某个情节元素的独特程度。通过对100篇短篇故事进行评估，我们发现LLM生成的故事常常包含特殊情节元素的组合，这些组合在多次生成过程中、甚至在不同LLM之间都频繁重复出现；而人类原创故事的情节则极少被复现，甚至连零散的情节片段都很少被重复。此外，我们的人工评估显示，尽管Sui Generis分数的计算完全自动、不依赖人工判断，但故事片段的Sui Generis分数排名与人类对惊喜程度的判断呈中等相关性。
英文摘要：
With rapid advances in large language models (LLMs), there has been an increasing application of LLMs in creative content ideation and generation. A critical question emerges: can current LLMs provide ideas that are diverse enough to truly bolster collective creativity? We examine two state-of-the-art LLMs, GPT-4 and LLaMA-3, on story generation and discover that LLM-generated stories often consist of plot elements that are echoed across a number of generations. To quantify this phenomenon, we introduce the Sui Generis score, an automatic metric that measures the uniqueness of a plot element among alternative storylines generated using the same prompt under an LLM. Evaluating on 100 short stories, we find that LLM-generated stories often contain combinations of idiosyncratic plot elements echoed frequently across generations and across different LLMs, while plots from the original human-written stories are rarely recreated or even echoed in pieces. Moreover, our human evaluation shows that the ranking of Sui Generis scores among story segments correlates moderately with human judgment of surprise level, even though score computation is completely automatic without relying on human judgment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
