arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2212.10013cs.AIcs.CL
收藏

DocAsRef:关于将基于参考的摘要质量评估指标无参考地重新利用的实证研究

DocAsRef: An Empirical Study on Repurposing Reference-Based Summary Quality Metrics Reference-Freely

Iowa State University(爱荷华州立大学);University of Wisconsin–Madison(威斯康星大学麦迪逊分校);ByteDance(字节跳动);East China Normal University(华东师范大学)

Forrest Sheng Bao, Ruixuan Tu, Ge Luo, Yinfei Yang, Hebi Li, Minghui Qiu, Youbiao He, Cen Chen

AI总结:

本文提出将基于参考的摘要质量评估指标改造为无参考指标,实验证明使用DeBERTa-large-MNLI的零样本BERTScore在SummEval和Newsroom数据集上优于原版及多数无参考指标,媲美GPT-3.5。

摘要:

自动摘要质量评估分为基于参考的和无参考的两类。基于参考的指标因人工参考提供的额外信息在历史上被认为更准确,但受限于对人工输入的依赖。本文假设,某些基于参考的指标用于评估系统摘要与其对应参考的比较方法,可被有效改造为针对源文档进行评估,从而将这些指标转化为无参考指标。实验结果支持该假设。在无参考改造后,使用参数量<0.5B的预训练DeBERTa-large-MNLI模型的零样本BERTScore,在SummEval和Newsroom数据集上的多个方面均持续优于其原始基于参考的版本。它也优于大多数现有无参考指标,并与基于GPT-3.5的零样本摘要评估器表现相当。

英文摘要:

Automated summary quality assessment falls into two categories: reference-based and reference-free. Reference-based metrics, historically deemed more accurate due to the additional information provided by human-written references, are limited by their reliance on human input. In this paper, we hypothesize that the comparison methodologies used by some reference-based metrics to evaluate a system summary against its corresponding reference can be effectively adapted to assess it against its source document, thereby transforming these metrics into reference-free ones. Experimental results support this hypothesis. After being repurposed reference-freely, the zero-shot BERTScore using the pretrained DeBERTa-large-MNLI model of <0.5B parameters consistently outperforms its original reference-based version across various aspects on the SummEval and Newsroom datasets. It also excels in comparison to most existing reference-free metrics and closely competes with zero-shot summary evaluators based on GPT-3.5.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑