理解摘要中的事实性错误：错误、摘要器、数据集、错误检测器
Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectors
AI总结：
本文汇总九个数据集的事实性错误标注并按摘要模型分层，评估现有事实性检测指标，发现其性能因模型和错误类型而异，近期改进主要来自旧模型摘要，并据此提出最佳实践建议。
摘要：
抽象式摘要模型产生事实性错误的倾向已被广泛研究，包括设计用于检测事实性错误的指标，以及对当前系统输出中的错误进行标注。然而，摘要系统、指标和标注基准不断演变的特性使事实性评估成为一个移动目标，而在各指标之间进行清晰比较已变得越来越困难。在本工作中，我们汇总了来自九个现有数据集的事实性错误标注，并根据其底层的摘要模型对它们进行分层。我们在这个分层基准上比较了最先进的事实性指标（包括近期基于ChatGPT的指标）的性能，并表明它们的性能在不同类型的摘要模型之间存在显著差异。关键的是，我们的分析表明，事实性检测领域近期的改进大多来自对较旧（Transformer之前）模型生成的摘要，而不是更相关的近期摘要模型。我们进一步按错误类型进行了更细粒度的分析，发现不同事实性指标在不同错误类型上存在类似的性能差异。我们的结果表明，没有任何一个指标在所有设置或所有错误类型中都占优，基于这些见解，我们为最佳实践提供了建议。
英文摘要：
The propensity of abstractive summarization models to make factual errors has been studied extensively, including design of metrics to detect factual errors and annotation of errors in current systems' outputs. However, the ever-evolving nature of summarization systems, metrics, and annotated benchmarks makes factuality evaluation a moving target, and drawing clear comparisons among metrics has become increasingly difficult. In this work, we aggregate factuality error annotations from nine existing datasets and stratify them according to the underlying summarization model. We compare performance of state-of-the-art factuality metrics, including recent ChatGPT-based metrics, on this stratified benchmark and show that their performance varies significantly across different types of summarization models. Critically, our analysis shows that much of the recent improvement in the factuality detection space has been on summaries from older (pre-Transformer) models instead of more relevant recent summarization models. We further perform a finer-grained analysis per error-type and find similar performance variance across error types for different factuality metrics. Our results show that no one metric is superior in all settings or for all error types, and we provide recommendations for best practices given these insights.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
