评估用于小说生成的语言模型的世界观
Assessing Language Models' Worldview for Fiction Generation
AI总结：
该研究针对大语言模型在小说生成场景下的世界观维持能力展开评估，发现多数模型存在自我冲突、叙事模式趋同的问题，指出其缺乏小说生成必需的世界状态，并提出相关研究方向。
摘要：
大语言模型（LLMs）的应用已十分普遍，在计算创意领域有着丰富的应用场景，小说故事生成便是其中之一。小说是发生在与现实世界略有差异的故事世界中的叙事。随着LLMs逐渐成为写作搭档，我们质疑它们生成小说的适配性。本研究探究了LLMs维持小说生成所必需的世界状态的能力。通过向9个LLMs提出一系列问题，我们发现仅有2个模型表现出一致的世界观，其余模型均存在自我冲突。随后对4个模型生成的故事进行的分析显示，它们的叙事模式惊人地统一。这种跨模型的统一性进一步表明LLMs缺乏小说所需的“状态”。我们强调了当前LLMs在小说写作方面的局限性，并倡导未来研究测试并构建可供LLMs依托的故事世界。所有代码、数据集和生成的回复可在https://github.com/tanny411/llm-reliability-and-consistency-evaluation获取。
英文摘要：
The use of Large Language Models (LLMs) has become ubiquitous, with abundant applications in computational creativity. One such application is fictional story generation. Fiction is a narrative that occurs in a story world that is slightly different than ours. With LLMs becoming writing partners, we question how suitable they are to generate fiction. This study investigates the ability of LLMs to maintain a state of world essential to generate fiction. Through a series of questions to nine LLMs, we find that only two models exhibit consistent worldview, while the rest are self-conflicting. Subsequent analysis of stories generated by four models revealed a strikingly uniform narrative pattern. This uniformity across models further suggests a lack of `state' necessary for fiction. We highlight the limitations of current LLMs in fiction writing and advocate for future research to test and create story worlds for LLMs to reside in. All code, dataset, and the generated responses can be found in https://github.com/tanny411/llm-reliability-and-consistency-evaluation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
