arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2305.16895cs.CLcs.AI
收藏

UMSE:统一多场景摘要评估

UMSE: Unified Multi-scenario Summarization Evaluation

Shandong University(山东大学);Microsoft Corporation(微软公司);King Abdullah University of Science and Technology(阿卜杜拉国王科技大学)

Shen Gao, Zhitao Yao, Chongyang Tao, Xiuying Chen, Pengjie Ren, Zhaochun Ren, Zhumin Chen

AI总结:

本文提出统一多场景摘要评估模型UMSE,通过扰动前缀调优和自监督训练共享跨场景知识,首次实现单一框架同时支持基于参考、无参考等多种摘要评估场景,并在SummEval上取得与专用强方法相当的性能。

摘要:

摘要质量评估是文本摘要中一项非平凡的任务。当代方法主要可分为两种场景:(1)基于参考的评估:利用人工标注的参考摘要进行评估;(2)无参考评估:评估文档的摘要一致性。近期研究主要关注其中一种场景,并探索构建基于预训练语言模型(PLMs)的神经模型以对齐人类标准。然而,不同场景的模型是分别优化的,这可能导致次优性能,因为它们忽略了跨场景的共享知识。此外,为每个场景设计单独的模型也给用户带来了不便。受此启发,我们提出了统一多场景摘要评估模型(UMSE)。具体而言,我们提出了一种扰动前缀调优方法,以在场景之间共享跨场景知识,并使用自监督训练范式在无需额外人工标注的情况下优化模型。我们的UMSE是第一个具备在三种评估场景中使用能力的统一摘要评估框架。在基准数据集SummEval上跨三种典型场景的实验结果表明,我们的UMSE能够取得与若干专门为每个场景设计的现有强方法相当的性能。

英文摘要:

Summarization quality evaluation is a non-trivial task in text summarization. Contemporary methods can be mainly categorized into two scenarios: (1) reference-based: evaluating with human-labeled reference summary; (2) reference-free: evaluating the summary consistency of the document. Recent studies mainly focus on one of these scenarios and explore training neural models built on PLMs to align with human criteria. However, the models from different scenarios are optimized individually, which may result in sub-optimal performance since they neglect the shared knowledge across different scenarios. Besides, designing individual models for each scenario caused inconvenience to the user. Inspired by this, we propose Unified Multi-scenario Summarization Evaluation Model (UMSE). More specifically, we propose a perturbed prefix tuning method to share cross-scenario knowledge between scenarios and use a self-supervised training paradigm to optimize the model without extra human labeling. Our UMSE is the first unified summarization evaluation framework engaged with the ability to be used in three evaluation scenarios. Experimental results across three typical scenarios on the benchmark dataset SummEval indicate that our UMSE can achieve comparable performance with several existing strong methods which are specifically designed for each scenario.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑