arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2006.11880cs.CLcs.AIcs.LG
收藏

机器阅读理解研究综述:任务、评估模型和基准数据集

A Survey on Machine Reading Comprehension: Tasks, Evaluation Metrics and Benchmark Datasets

Chengdu Institute of Computer Application, Chinese Academy of Sciences(中国科学院成都计算机应用研究所);University of Chinese Academy of Sciences(中国科学院大学);Guizhou University(贵州大学);GuiZhou University of Finance and Economics(贵州财经大学);University of South Carolina(南卡罗来纳大学);Chengdu Neusoft University(成都东软学院)

Changchang Zeng, Shaobo Li, Qin Li, Jie Hu, Jianjun Hu

摘要:

机器阅读理解(MRC)是自然语言处理(NLP)中一个具有挑战性的研究领域,具有广泛的实际应用。近年来该领域的巨大进步主要得益于大规模数据集和深度学习的出现。目前,许多MRC模型已经在各种基准数据集上超过了人类的表现,尽管现有的MRC模型与真正的人类水平的阅读理解之间存在明显的巨大差距。这表明需要改进现有的数据集,评估指标和模型,以使当前的MRC模型走向“真正的”理解。针对现有MRC任务、评价指标和数据集缺乏全面综述的现状,本文(1)分析了57个MRC任务和数据集,提出了4种不同属性的MRC任务的更精确分类方法;(2)总结了MRC任务的9个评价指标、MRC数据集的7个属性和10个特征;(3)讨论了MRC研究中存在的主要问题,并指出了今后的研究方向。此外,我们还在配套网站(https://mrc-martets.github.io/)上收集、组织和发布了我们的数据,MRC研究人员可以直接访问每个MRC数据集、论文、基线项目和排行榜。

英文摘要:

Machine Reading Comprehension (MRC) is a challenging Natural Language Processing(NLP) research field with wide real-world applications. The great progress of this field in recent years is mainly due to the emergence of large-scale datasets and deep learning. At present, a lot of MRC models have already surpassed human performance on various benchmark datasets despite the obvious giant gap between existing MRC models and genuine human-level reading comprehension. This shows the need for improving existing datasets, evaluation metrics, and models to move current MRC models toward "real" understanding. To address the current lack of comprehensive survey of existing MRC tasks, evaluation metrics, and datasets, herein, (1) we analyze 57 MRC tasks and datasets and propose a more precise classification method of MRC tasks with 4 different attributes; (2) we summarized 9 evaluation metrics of MRC tasks, 7 attributes and 10 characteristics of MRC datasets; (3) We also discuss key open issues in MRC research and highlighted future research directions. In addition, we have collected, organized, and published our data on the companion website(https://mrc-datasets.github.io/) where MRC researchers could directly access each MRC dataset, papers, baseline projects, and the leaderboard.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑