arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2010.07100cs.CLcs.IRcs.LG
收藏

文本摘要评价的再认识

Re-evaluating Evaluation in Text Summarization

Carnegie Mellon University(卡内基梅隆大学)

Manik Bhandari, Pranav Gour, Atabak Ashfaq, Pengfei Liu, Graham Neubig

摘要:

自动评估指标作为手动评估的替代,是文本生成任务(如文本摘要)开发的重要组成部分。然而,虽然该领域已经取得了进展,但我们的标准指标却没有-近20年来,ROUGE一直是大多数摘要论文的标准评估。在本文中,我们试图重新评估文本摘要的评估方法:使用最高评分的系统输出,抽象和提取,最近流行的系统级和摘要级评估设置的数据集,评估自动度量的可靠性。我们发现,关于旧数据集的评估指标的结论不一定适用于现代数据集和系统。

英文摘要:

Automated evaluation metrics as a stand-in for manual evaluation are an essential part of the development of text-generation tasks such as text summarization. However, while the field has progressed, our standard metrics have not -- for nearly 20 years ROUGE has been the standard evaluation in most summarization papers. In this paper, we make an attempt to re-evaluate the evaluation method for text summarization: assessing the reliability of automatic metrics using top-scoring system outputs, both abstractive and extractive, on recently popular datasets for both system-level and summary-level evaluation settings. We find that conclusions about evaluation metrics on older datasets do not necessarily hold on modern datasets and systems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑