arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2212.13428cs.CL
收藏

知识增强的预训练语言模型研究综述

A Survey on Knowledge-Enhanced Pre-trained Language Models

Beijing University of Posts and Telecommunications(北京邮电大学);Thomson Reuters Labs(汤森路透实验室)

Chaoqi Zhen, Yanlei Shang, Xiangyu Liu, Yifei Li, Yong Chen, Dell Zhang

摘要:

自然语言处理(NLP)通过使用预训练语言模型(PLM)(如BERT)而发生了革命性的变化。尽管在几乎每一个NLP任务中都创造了新的记录,但PLM仍然面临着许多挑战,包括可解释性差,推理能力弱,以及在应用于下游任务时需要大量昂贵的注释数据。通过将外部知识集成到PLM中,\textit{\underline{K}nowledge-\underline{E}nhanced \underline{P}re-trained \underline{L}anguage \underline{M}odels}(KEPLM)有可能克服上述局限性。在本文中,我们通过一系列的研究系统地检查KEPLMs。具体来说,我们概述了常见的类型和不同格式的知识集成到KEPLMs,详细介绍了现有的方法,用于构建和评估KEPLMS,目前的应用程序KEPLMs在下游任务,并讨论了未来的研究方向。研究人员将从这项调查中受益,获得该领域最新发展的快速和全面的概述。

英文摘要:

Natural Language Processing (NLP) has been revolutionized by the use of Pre-trained Language Models (PLMs) such as BERT. Despite setting new records in nearly every NLP task, PLMs still face a number of challenges including poor interpretability, weak reasoning capability, and the need for a lot of expensive annotated data when applied to downstream tasks. By integrating external knowledge into PLMs, \textit{\underline{K}nowledge-\underline{E}nhanced \underline{P}re-trained \underline{L}anguage \underline{M}odels} (KEPLMs) have the potential to overcome the above-mentioned limitations. In this paper, we examine KEPLMs systematically through a series of studies. Specifically, we outline the common types and different formats of knowledge to be integrated into KEPLMs, detail the existing methods for building and evaluating KEPLMS, present the applications of KEPLMs in downstream tasks, and discuss the future research directions. Researchers will benefit from this survey by gaining a quick and comprehensive overview of the latest developments in this field.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑