对长文档摘要的事实一致性度量进行压力测试
Stress Testing Factual Consistency Metrics for Long-Document Summarization
AI总结：
本文评估了六个常用无参考事实性度量在长文档摘要中的可靠性，发现短形式度量在处理信息密集型声明时可靠性下降，提出改进方向包括多跨度推理、上下文感知校准和训练保留意义的变体。
摘要：
评估抽象文本摘要的事实一致性仍是一个重大挑战，尤其是在长文档中，传统度量在输入长度限制和长程依赖性方面表现不佳。本文系统评估了六个广泛使用的无参考事实性度量在长文档设置中的可靠性。我们通过七种事实性保持扰动对摘要进行测试，包括改写、简化、同义词替换、逻辑等价否定、词汇减少、压缩和源文本插入，并进一步分析其对检索上下文和声明信息密度的敏感性。在涵盖科幻、法律和科学领域的三个长形式基准数据集中，我们的结果表明，现有短形式度量对语义等价摘要产生不一致评分，并且在信息密集型声明的可靠性下降，这些声明内容与源文档的许多部分语义相似。虽然扩展检索上下文在某些领域提高了稳定性，但没有度量在长上下文条件下保持事实一致性。最后，我们的结果强调了改进事实性评估的具体方向，包括多跨度推理、上下文感知校准和在保留意义的变体上训练以增强长形式摘要的鲁棒性。我们发布了所有代码、扰动数据和所需的脚本，以在 https://github.com/zainmujahid/metricEval-longSum 复现我们的结果。
英文摘要：
Evaluating the factual consistency of abstractive text summarization remains a significant challenge, particularly for long documents, where conventional metrics struggle with input length limitations and long-range dependencies. In this work, we systematically evaluate the reliability of six widely used reference-free factuality metrics, originally proposed for short-form summarization, in the long-document setting. We probe metric robustness through seven factuality-preserving perturbations applied to summaries, namely paraphrasing, simplification, synonym replacement, logically equivalent negations, vocabulary reduction, compression, and source text insertion, and further analyze their sensitivity to retrieval context and claim information density. Across three long-form benchmark datasets spanning science fiction, legal, and scientific domains, our results reveal that existing short-form metrics produce inconsistent scores for semantically equivalent summaries and exhibit declining reliability for information-dense claims whose content is semantically similar to many parts of the source document. While expanding the retrieval context improves stability in some domains, no metric consistently maintains factual alignment under long-context conditions. Finally, our results highlight concrete directions for improving factuality evaluation, including multi-span reasoning, context-aware calibration, and training on meaning-preserving variations to enhance robustness in long-form summarization. We release all code, perturbed data, and scripts required to reproduce our results at https://github.com/zainmujahid/metricEval-longSum.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
