不太可能的对决：通过独特场景评估大语言模型的创意写作能力
The Unlikely Duel: Evaluating Creative Writing in LLMs through a Unique Scenario
AI总结：
该研究通过伊格内修斯与翼手龙对决的定制提示词评估多款指令微调LLM的创意写作能力，发现商业前沿LLM多数维度持平或略超人类，开源模型落后，人类原创性领先，仅前三LLM幽默表现达人类水平。
摘要：
本文是发表于EMNLP 2023 Findings的论文《模型联盟：大语言模型创意写作综合评估》的概要。我们针对一系列最新的指令微调前沿大语言模型（LLMs）开展英文创意写作任务评估，并将其与人类作者进行对比。为此，我们使用了一个专门设计的提示词（以约翰·肯尼迪·图尔《笨蛋联盟》主角伊格内修斯·J·赖利与翼手龙的史诗级对决为蓝本），以降低训练数据泄露风险，倒逼模型发挥创意而非复用现有故事。我们向LLMs和人类作者提供相同提示词，由人类依据包含流畅度、风格、原创性、幽默性等多维度的详细评分标准进行评估。结果显示，部分前沿商业LLMs在多数评估维度上与人类作者表现相当或略胜一筹，开源LLMs则表现落后。人类在原创性上仍保持微弱领先，且仅排名前三的LLMs能达到类人类的幽默处理水平。
英文摘要：
This is a summary of the paper "A Confederacy of Models: a Comprehensive Evaluation of LLMs on Creative Writing", which was published in Findings of EMNLP 2023. We evaluate a range of recent state-of-the-art, instruction-tuned large language models (LLMs) on an English creative writing task, and compare them to human writers. For this purpose, we use a specifically-tailored prompt (based on an epic combat between Ignatius J. Reilly, main character of John Kennedy Toole's "A Confederacy of Dunces", and a pterodactyl) to minimize the risk of training data leakage and force the models to be creative rather than reusing existing stories. The same prompt is presented to LLMs and human writers, and evaluation is performed by humans using a detailed rubric including various aspects like fluency, style, originality or humor. Results show that some state-of-the-art commercial LLMs match or slightly outperform our human writers in most of the evaluated dimensions. Open-source LLMs lag behind. Humans keep a close lead in originality, and only the top three LLMs can handle humor at human-like levels.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
