arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.05669cs.CL
收藏

ConcEPT:面向语言模型的概念增强预训练

ConcEPT: Concept-Enhanced Pre-Training for Language Models

Fudan University(复旦大学);Fudan-Aishu Cognitive Intelligence Joint Research Center(复旦-爱数认知智能联合研究中心)

Xintao Wang, Zhouhong Gu, Jiaqing Liang, Dakuan Lu, Yanghua Xiao, Wei Wang

AI总结:

本文提出ConcEPT,通过实体概念预测预训练目标将外部分类体系中的概念知识注入预训练语言模型,无需实体链接或概念映射即可适配下游任务,并在实体分型等四个任务上验证了其有效性。

摘要:

预训练语言模型(PLMs)已在自然语言处理的最先进方法中占据主导地位,并且知识增强型PLMs被进一步提出以提升模型在知识密集型任务上的性能。然而,概念知识作为人类认知中一种必不可少的知识类型,在这类研究中仍未得到充分探索。这限制了PLMs在需要类人认知的场景中的表现,例如理解带有概念的长尾实体。在本文中,我们提出ConcEPT,即面向语言模型的概念增强预训练,用于将概念知识注入PLMs。ConcEPT利用外部分类体系进行实体概念预测,这是一种新颖的预训练目标,用于预测预训练上下文中提及实体的概念。与以往的概念增强方法不同,ConcEPT可以无需实体链接或概念映射即可便捷地适配各种下游应用。大量实验结果表明ConcEPT在实体分型等四个任务上的有效性,验证了我们的模型通过概念增强预训练获得了改进的概念知识。

英文摘要:

Pre-trained language models (PLMs) have been prevailing in state-of-the-art methods for natural language processing, and knowledge-enhanced PLMs are further proposed to promote model performance in knowledge-intensive tasks. However, conceptual knowledge, one essential kind of knowledge for human cognition, still remains understudied in this line of research. This limits PLMs' performance in scenarios requiring human-like cognition, such as understanding long-tail entities with concepts. In this paper, we propose ConcEPT, which stands for Concept-Enhanced Pre-Training for language models, to infuse conceptual knowledge into PLMs. ConcEPT exploits external taxonomies with entity concept prediction, a novel pre-training objective to predict the concepts of entities mentioned in the pre-training contexts. Unlike previous concept-enhanced methods, ConcEPT can be readily adapted to various downstream applications without entity linking or concept mapping. Results of extensive experiments show the effectiveness of ConcEPT in four tasks such as entity typing, which validates that our model gains improved conceptual knowledge with concept-enhanced pre-training.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑