References Matter：探究参考集变化对摘要评估的影响
References Matter: Investigating the Impact of Reference Set Variation on Summarization Evaluation
AI总结：
本研究系统调查了参考集变化对摘要评估指标的影响，发现ROUGE等流行指标存在显著不稳定性，且与LLM输出的人类判断相关性微弱，建议将参考集变化纳入评估以增强一致性。
摘要：
人类语言产出表现出显著的丰富性和多样性，反映了不同的交流风格和意图。然而，这种多样性在摘要评估中常被忽视。尽管已知拥有多个参考摘要能提高与人类判断的相关性，但参考集对基于参考的指标的影响尚未得到系统调查。本研究考察了广泛使用的基于参考的指标对参考集选择的敏感性，分析了三个多样化的多参考摘要数据集：SummEval、GUMSum和DUC2004。我们证明，许多流行的指标表现出显著的不稳定性。这种不稳定性对于像ROUGE这样基于n-gram的指标尤为令人担忧，因为模型排名会根据参考集的不同而变化，从而破坏了模型比较的可靠性。我们还收集了针对体裁多样化数据的LLM输出的人类判断，并考察了它们与指标的相关性，以补充新闻报道摘要之外的现有发现，发现相关性微弱甚至没有相关性。综上所述，我们建议将参考集变化纳入摘要评估，以增强一致性以及与人类判断的相关性，特别是在评估LLM时。
英文摘要：
Human language production exhibits remarkable richness and variation, reflecting diverse communication styles and intents. However, this variation is often overlooked in summarization evaluation. While having multiple reference summaries is known to improve correlation with human judgments, the impact of the reference set on reference-based metrics has not been systematically investigated. This work examines the sensitivity of widely used reference-based metrics in relation to the choice of reference sets, analyzing three diverse multi-reference summarization datasets: SummEval, GUMSum, and DUC2004. We demonstrate that many popular metrics exhibit significant instability. This instability is particularly concerning for n-gram-based metrics like ROUGE, where model rankings vary depending on the reference sets, undermining the reliability of model comparisons. We also collect human judgments on LLM outputs for genre-diverse data and examine their correlation with metrics to supplement existing findings beyond newswire summaries, finding weak-to-no correlation. Taken together, we recommend incorporating reference set variation into summarization evaluation to enhance consistency alongside correlation with human judgments, especially when evaluating LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
