arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2109.09195cs.CL
收藏

研究众包协议以评估摘要的事实一致性

Investigating Crowdsourcing Protocols for Evaluating the Factual Consistency of Summaries

Yale University(耶鲁大学);Columbia University(哥伦比亚大学);Facebook AI(脸书人工智能)

Xiangru Tang, Alexander Fabbri, Haoran Li, Ziming Mao, Griffin Thomas Adams, Borui Wang, Asli Celikyilmaz, Yashar Mehdad, Dragomir Radev

摘要:

当前应用于摘要的预训练模型容易出现事实不一致,要么歪曲源文本,要么引入无关信息。因此,在我们开发改进的模型时,比较摘要的事实一致性是必要的。然而,事实一致性的最佳人类评估设置尚未标准化。为了解决这个问题,我们使用基于评级的李克特量表和基于排名的最佳-最差缩放协议,对来自CNN-每日邮报和XSum数据集的100篇文章进行了众包评估,以确定最可靠的评估框架。我们发现,基于排名的协议提供了一个更可靠的衡量汇总质量的数据集,而李克特评级的可靠性取决于目标数据集和评估设计。我们的众包模板和摘要评估将公开提供,以促进未来对摘要中事实一致性的研究。

英文摘要:

Current pre-trained models applied to summarization are prone to factual inconsistencies which either misrepresent the source text or introduce extraneous information. Thus, comparing the factual consistency of summaries is necessary as we develop improved models. However, the optimal human evaluation setup for factual consistency has not been standardized. To address this issue, we crowdsourced evaluations for factual consistency using the rating-based Likert scale and ranking-based Best-Worst Scaling protocols, on 100 articles from each of the CNN-Daily Mail and XSum datasets over four state-of-the-art models, to determine the most reliable evaluation framework. We find that ranking-based protocols offer a more reliable measure of summary quality across datasets, while the reliability of Likert ratings depends on the target dataset and the evaluation design. Our crowdsourcing templates and summary evaluations will be publicly available to facilitate future research on factual consistency in summarization.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑