DocAsRef:关于将基于参考的摘要质量评估指标无参考地重新利用的实证研究
DocAsRef: An Empirical Study on Repurposing Reference-Based Summary Quality Metrics Reference-Freely
AI总结:
本文提出将基于参考的摘要质量评估指标改造为无参考指标,实验证明使用DeBERTa-large-MNLI的零样本BERTScore在SummEval和Newsroom数据集上优于原版及多数无参考指标,媲美GPT-3.5。
摘要:
自动摘要质量评估分为基于参考的和无参考的两类。基于参考的指标因人工参考提供的额外信息在历史上被认为更准确,但受限于对人工输入的依赖。本文假设,某些基于参考的指标用于评估系统摘要与其对应参考的比较方法,可被有效改造为针对源文档进行评估,从而将这些指标转化为无参考指标。实验结果支持该假设。在无参考改造后,使用参数量<0.5B的预训练DeBERTa-large-MNLI模型的零样本BERTScore,在SummEval和Newsroom数据集上的多个方面均持续优于其原始基于参考的版本。它也优于大多数现有无参考指标,并与基于GPT-3.5的零样本摘要评估器表现相当。
英文摘要:
Automated summary quality assessment falls into two categories: reference-based and reference-free. Reference-based metrics, historically deemed more accurate due to the additional information provided by human-written references, are limited by their reliance on human input. In this paper, we hypothesize that the comparison methodologies used by some reference-based metrics to evaluate a system summary against its corresponding reference can be effectively adapted to assess it against its source document, thereby transforming these metrics into reference-free ones. Experimental results support this hypothesis. After being repurposed reference-freely, the zero-shot BERTScore using the pretrained DeBERTa-large-MNLI model of <0.5B parameters consistently outperforms its original reference-based version across various aspects on the SummEval and Newsroom datasets. It also excels in comparison to most existing reference-free metrics and closely competes with zero-shot summary evaluators based on GPT-3.5.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。