arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2010.00796cs.CL
收藏

JAKET:知识图谱和语言理解的联合预训练

JAKET: Joint Pre-training of Knowledge Graph and Language Understanding

Carnegie Mellon University(卡内基梅隆大学);Microsoft Cognitive Services Research Group(微软认知服务研究组)

Donghan Yu, Chenguang Zhu, Yiming Yang, Michael Zeng

摘要:

知识图谱包含了丰富的世界知识、实体和关系信息。因此,它们可以成为现有预训练语言模型的重要补充。然而,它仍然是一个挑战,有效地将信息从KG语言建模。而对知识图谱的理解需要相关的语境。我们提出了一个新的联合预训练框架,JAKET,模型的知识图谱和语言。知识模块和语言模块提供相互帮助的基本信息:知识模块为文本中的实体生成嵌入,而语言模块为图中的实体和关系生成上下文感知的初始嵌入。我们的设计使预训练模型能够轻松适应新领域中看不见的知识图谱。几个知识感知的自然语言处理任务的实验结果表明,我们提出的框架实现了优越的性能,有效地利用知识的语言理解。

英文摘要:

Knowledge graphs (KGs) contain rich information about world knowledge, entities and relations. Thus, they can be great supplements to existing pre-trained language models. However, it remains a challenge to efficiently integrate information from KG into language modeling. And the understanding of a knowledge graph requires related context. We propose a novel joint pre-training framework, JAKET, to model both the knowledge graph and language. The knowledge module and language module provide essential information to mutually assist each other: the knowledge module produces embeddings for entities in text while the language module generates context-aware initial embeddings for entities and relations in the graph. Our design enables the pre-trained model to easily adapt to unseen knowledge graphs in new domains. Experimental results on several knowledge-aware NLP tasks show that our proposed framework achieves superior performance by effectively leveraging knowledge in language understanding.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑