帮我写一个故事：评估大语言模型生成写作反馈的能力
Help Me Write a Story: Evaluating LLMs' Ability to Generate Writing Feedback
AI总结：
本文通过构建包含1,300个故意引入写作问题的故事测试集及评估框架，研究大语言模型生成写作反馈的能力，发现模型能提供具体且大多准确的反馈，但常无法识别最大写作问题并正确区分批评性与正面反馈的时机。
摘要：
大语言模型能否通过提供有意义的写作反馈来支持创意写作者？在本文中，我们通过定义一个新任务、数据集和评估框架，探讨了模型生成写作反馈所面临的挑战与局限。为了以受控方式研究模型性能，我们提出了一个包含1,300个故事的新测试集，并故意对这些故事进行破坏以引入写作问题。我们使用自动评估指标和人工评估指标，研究了常用大语言模型在该任务上的表现。我们的分析表明，当前模型在许多方面表现出强大的开箱即用行为——能够提供具体且大多准确的写作反馈。然而，模型往往无法识别故事中最大的写作问题，也无法正确判断何时应提供批评性反馈还是正面反馈。
英文摘要：
Can LLMs provide support to creative writers by giving meaningful writing feedback? In this paper, we explore the challenges and limitations of model-generated writing feedback by defining a new task, dataset, and evaluation frameworks. To study model performance in a controlled manner, we present a novel test set of 1,300 stories that we corrupted to intentionally introduce writing issues. We study the performance of commonly used LLMs in this task with both automatic and human evaluation metrics. Our analysis shows that current models have strong out-of-the-box behavior in many respects -- providing specific and mostly accurate writing feedback. However, models often fail to identify the biggest writing issue in the story and to correctly decide when to offer critical vs. positive feedback.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
