FABLES：评估书籍长度摘要中的忠实性与内容选择
FABLES: Evaluating faithfulness and content selection in book-length summarization
AI总结：
本文通过构建FABLES数据集首次对书籍长度摘要进行大规模人工评估，发现Claude-3-Opus忠实性最优，并揭示现有LLM评分器无法有效检测不忠实声明及内容选择中的系统性偏差。
摘要：
尽管长上下文大语言模型（LLMs）在技术上能够对书籍长度的文档（超过10万token）进行摘要，但文档的长度和复杂性迄今阻碍了对输入依赖方面（如忠实性）的评估。在本文中，我们首次对虚构类书籍的LLM生成摘要进行了大规模的忠实性与内容选择人工评估。我们的研究通过聚焦于2023年或2024年出版的书籍摘要来缓解数据污染问题，并且我们聘请在标注任务之前已完整阅读每本书籍的标注者，以最小化成本和认知负担。我们收集了FABLES数据集，该数据集包含对26本书籍的LLM生成摘要中3,158条声明的人工标注，成本为5,200美元，这使我们能够基于忠实性对LLM摘要器进行排名：Claude-3-Opus显著优于所有闭源LLM，而开源的Mixtral与GPT-3.5-Turbo不相上下。对标注的分析表明，大多数不忠实的声明与事件和角色状态有关，并且通常需要对叙事进行间接推理才能判定其无效。尽管基于LLM的自动评分器在其他场景中已被证明对事实性和连贯性可靠，但我们实现了多个忠实性LLM评分器，并发现没有一个与人工标注强相关，尤其是在检测不忠实声明方面。我们的实验表明，检测不忠实声明不仅是摘要评估的重要未来方向，也是长上下文理解的测试平台。最后，我们超越忠实性，探索了书籍长度摘要中的内容选择错误：我们开发了一种与关键叙事元素相关的遗漏错误类型学，并识别出一种系统性的对发生在书籍结尾附近事件的过度强调。
英文摘要：
While long-context large language models (LLMs) can technically summarize book-length documents (>100K tokens), the length and complexity of the documents have so far prohibited evaluations of input-dependent aspects like faithfulness. In this paper, we conduct the first large-scale human evaluation of faithfulness and content selection on LLM-generated summaries of fictional books. Our study mitigates the issue of data contamination by focusing on summaries of books published in 2023 or 2024, and we hire annotators who have fully read each book prior to the annotation task to minimize cost and cognitive burden. We collect FABLES, a dataset of annotations on 3,158 claims made in LLM-generated summaries of 26 books, at a cost of $5.2K USD, which allows us to rank LLM summarizers based on faithfulness: Claude-3-Opus significantly outperforms all closed-source LLMs, while the open-source Mixtral is on par with GPT-3.5-Turbo. An analysis of the annotations reveals that most unfaithful claims relate to events and character states, and they generally require indirect reasoning over the narrative to invalidate. While LLM-based auto-raters have proven reliable for factuality and coherence in other settings, we implement several LLM raters of faithfulness and find that none correlates strongly with human annotations, especially with regard to detecting unfaithful claims. Our experiments suggest that detecting unfaithful claims is an important future direction not only for summarization evaluation but also as a testbed for long-context understanding. Finally, we move beyond faithfulness by exploring content selection errors in book-length summarization: we develop a typology of omission errors related to crucial narrative elements and also identify a systematic over-emphasis on events occurring towards the end of the book.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
