arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2107.08582cs.CL
收藏

跨越语言模型与阅读理解的鸿沟--基于自我监督的无监督MRC

Bridging the Gap between Language Model and Reading Comprehension: Unsupervised MRC via Self-Supervision

University of Chinese Academy of Sciences(中国科学院大学);Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所)

Ning Bian, Xianpei Han, Bo Chen, Hongyu Lin, Ben He, Le Sun

摘要:

尽管最近在机器阅读理解(MRC)方面取得了成功,但学习高质量的MRC模型仍然需要大规模的标记训练数据,即使使用强大的预训练语言模型(PLM)。PLM的预训练任务不是问答或基于MRC的任务,使得现有的PLM无法直接用于无监督的MRC。具体来说,MRC的目标是从给定的文档中找到一个准确的答案,而PLM则专注于句子中的标记填充。在本文中,我们提出了一个新的框架,无监督MRC。首先,我们提出通过自监督学习来学习发现文档中的答案跨度,通过为MRC - Spotting-MLM设计一个自监督借口任务。解决这个任务需要捕捉文档中句子之间的深层交互。其次,我们在推理阶段采用简单的句子重写策略,以减轻问题和文档之间的表达不匹配。实验表明,我们的方法实现了一个新的国家的最先进的性能无监督MRC。

英文摘要:

Despite recent success in machine reading comprehension (MRC), learning high-quality MRC models still requires large-scale labeled training data, even using strong pre-trained language models (PLMs). The pre-training tasks for PLMs are not question-answering or MRC-based tasks, making existing PLMs unable to be directly used for unsupervised MRC. Specifically, MRC aims to spot an accurate answer span from the given document, but PLMs focus on token filling in sentences. In this paper, we propose a new framework for unsupervised MRC. Firstly, we propose to learn to spot answer spans in documents via self-supervised learning, by designing a self-supervision pretext task for MRC - Spotting-MLM. Solving this task requires capturing deep interactions between sentences in documents. Secondly, we apply a simple sentence rewriting strategy in the inference stage to alleviate the expression mismatch between questions and documents. Experiments show that our method achieves a new state-of-the-art performance for unsupervised MRC.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑