arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2106.11388cs.CLcs.LG
收藏

您对摘要数据集了解多少?

How well do you know your summarization datasets?

Carnegie Mellon University(卡内基梅隆大学)

Priyam Tejaswin, Dhruv Naik, Pengfei Liu

摘要:

最先进的摘要系统在从网络上抓取的大量数据集上进行训练和评估。尽管它们普遍存在,但我们对它们的基本特征(数据噪声,摘要复杂性等)知之甚少。这些数据集,以及它们如何影响系统性能和ROUGE等自动度量的可靠性。在这项研究中,我们手动分析了600个样本,从三个流行的摘要数据集。我们的研究是由六类类型学驱动的,它捕获了不同的噪音类型(缺失的事实,实体)和总结难度(提取,抽象)。我们随后对27种最先进的摘要模型和5种流行的指标进行了全面分析,并报告了我们的关键见解:(1)数据集具有不同的数据质量和复杂性分布,这可以追溯到它们的收集过程。(2)模型的性能和度量的可靠性取决于样本的复杂性。(3)由于参考文献的多样性差,忠实的摘要往往得分较低。我们发布代码、带注释的数据和模型输出。

英文摘要:

State-of-the-art summarization systems are trained and evaluated on massive datasets scraped from the web. Despite their prevalence, we know very little about the underlying characteristics (data noise, summarization complexity, etc.) of these datasets, and how these affect system performance and the reliability of automatic metrics like ROUGE. In this study, we manually analyze 600 samples from three popular summarization datasets. Our study is driven by a six-class typology which captures different noise types (missing facts, entities) and degrees of summarization difficulty (extractive, abstractive). We follow with a thorough analysis of 27 state-of-the-art summarization models and 5 popular metrics, and report our key insights: (1) Datasets have distinct data quality and complexity distributions, which can be traced back to their collection process. (2) The performance of models and reliability of metrics is dependent on sample complexity. (3) Faithful summaries often receive low scores because of the poor diversity of references. We release the code, annotated data and model outputs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑