arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2104.00054cs.CL
收藏

用Rescue法进行汇总评价的统计分析

A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methods

University of Pennsylvania(宾夕法尼亚大学)

Daniel Deutsch, Rotem Dror, Dan Roth

摘要:

摘要评价指标的质量通过计算其分数与大量摘要中的人工注释之间的相关性来量化。目前,还不清楚这些相关性估计的精确程度,也不清楚两个指标相关性之间的差异是否反映了真正的差异,或者这仅仅是偶然的。在这项工作中,我们解决这两个问题,提出的方法计算置信区间和运行假设检验的相关性使用两个restriking方法,自举和置换。在评估所提出的方法是最适合通过两个模拟实验总结后,我们分析了将这些方法应用于三组人类注释的几个不同的自动评估指标的结果。我们发现,置信区间是相当广泛的,表现出高度的不确定性,在自动度量的可靠性。此外,虽然许多指标未能显示ROUGE的统计改进,最近的两个作品,QAEval和BERTScore,在一些评估设置。

英文摘要:

The quality of a summarization evaluation metric is quantified by calculating the correlation between its scores and human annotations across a large number of summaries. Currently, it is unclear how precise these correlation estimates are, nor whether differences between two metrics' correlations reflect a true difference or if it is due to mere chance. In this work, we address these two problems by proposing methods for calculating confidence intervals and running hypothesis tests for correlations using two resampling methods, bootstrapping and permutation. After evaluating which of the proposed methods is most appropriate for summarization through two simulation experiments, we analyze the results of applying these methods to several different automatic evaluation metrics across three sets of human annotations. We find that the confidence intervals are rather wide, demonstrating high uncertainty in the reliability of automatic metrics. Further, although many metrics fail to show statistical improvements over ROUGE, two recent works, QAEval and BERTScore, do in some evaluation settings.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑