谨慎验证：依赖不完美事实性指标的陷阱
Verify with Caution: The Pitfalls of Relying on Imperfect Factuality Metrics
AI总结：
本文在11个数据集上重新评估了五种先进事实性指标，发现这些LLM评估器存在不一致性、系统性能误估及对改写和远距离信息的偏见，呼吁用户谨慎使用并手动验证其可靠性。
摘要：
大型语言模型的改进带来了日益增长的乐观情绪，认为它们可以作为自然语言生成输出的可靠评估器。在本文中，我们通过在涵盖摘要、检索增强生成和问答的11个数据集上全面重新评估五种最先进的事实性指标，来挑战这种乐观情绪。我们发现这些评估器彼此不一致，且经常错误估计系统级性能，这两者都可能导致各种陷阱。我们进一步表明，这些指标对高度改写的输出以及利用源文档较远部分的输出表现出偏见。我们敦促这些事实性指标的用户谨慎行事，并在继续之前手动验证这些指标在其感兴趣领域的可靠性。
英文摘要：
Improvements in large language models have led to increasing optimism that they can serve as reliable evaluators of natural language generation outputs. In this paper, we challenge this optimism by thoroughly re-evaluating five state-of-the-art factuality metrics on a collection of 11 datasets for summarization, retrieval-augmented generation, and question answering. We find that these evaluators are inconsistent with each other and often misestimate system-level performance, both of which can lead to a variety of pitfalls. We further show that these metrics exhibit biases against highly paraphrased outputs and outputs that draw upon faraway parts of the source documents. We urge users of these factuality metrics to proceed with caution and manually validate the reliability of these metrics in their domain of interest before proceeding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
