arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2210.08536cs.CL
收藏

面向自然语言理解的预训练语言模型中的知识表示

Knowledge Prompting in Pre-trained Language Model for Natural Language Understanding

East China Normal University(华东师范大学);Ant Group(蚂蚁集团);Alibaba Group(阿里巴巴集团)

Jianing Wang, Wenkang Huang, Qiuhui Shi, Hongbin Wang, Minghui Qiu, Xiang Li, Ming Gao

摘要:

知识增强的预训练语言模型(PLM)是近年来备受关注的一种语言模型,旨在将事实知识融入PLM中。然而,大多数现有的方法修改固定类型的PLM的内部结构,通过堆叠复杂的模块,并引入冗余和无关的事实知识库(KBs)。针对这些问题,本文提出了一种基于知识激励的PLM框架KP-PLM。该框架可以灵活地与现有的主流PLM相结合。具体来说,我们首先从每个上下文的知识库构建一个知识子图。然后设计多条连续提示规则,将知识子图转化为自然语言提示。为了进一步利用这些提示中的事实知识,我们提出了两个新的知识感知的自监督任务,包括提示相关性检查和掩蔽提示建模。对多个自然语言理解(NLU)任务的大量实验表明,KP-PLM在全资源和低资源环境中都优于其他最先进的方法。

英文摘要:

Knowledge-enhanced Pre-trained Language Model (PLM) has recently received significant attention, which aims to incorporate factual knowledge into PLMs. However, most existing methods modify the internal structures of fixed types of PLMs by stacking complicated modules, and introduce redundant and irrelevant factual knowledge from knowledge bases (KBs). In this paper, to address these problems, we introduce a seminal knowledge prompting paradigm and further propose a knowledge-prompting-based PLM framework KP-PLM. This framework can be flexibly combined with existing mainstream PLMs. Specifically, we first construct a knowledge sub-graph from KBs for each context. Then we design multiple continuous prompts rules and transform the knowledge sub-graph into natural language prompts. To further leverage the factual knowledge from these prompts, we propose two novel knowledge-aware self-supervised tasks including prompt relevance inspection and masked prompt modeling. Extensive experiments on multiple natural language understanding (NLU) tasks show the superiority of KP-PLM over other state-of-the-art methods in both full-resource and low-resource settings.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑