arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2009.13964cs.CLcs.AI
收藏

CokeBERT:面向增强预训练语言模型的上下文知识选择与嵌入

CokeBERT: Contextual Knowledge Selection and Embedding towards Enhanced Pre-Trained Language Models

Tsinghua University(清华大学);Tencent Inc.(腾讯公司)

Yusheng Su, Xu Han, Zhengyan Zhang, Peng Li, Zhiyuan Liu, Yankai Lin, Jie Zhou, Maosong Sun

AI总结:

提出CokeBERT框架,根据文本上下文动态选择并嵌入知识图谱中的知识,有效过滤冗余和模糊信息,在知识驱动NLP任务上优于基线方法且具有更好的可解释性。

摘要:

近期的一些研究致力于利用知识图谱(KGs)中的额外异构知识来增强预训练语言模型(PLMs),并在各种知识驱动的NLP任务上取得了持续的改进。然而,大多数知识增强的PLMs嵌入的是KG的静态子图(“知识上下文”),忽略了PLMs所需的知识可能会根据特定文本(“文本上下文”)动态变化。在本文中,我们提出了一个名为Coke的新框架,能够根据文本上下文为PLMs动态选择上下文知识并嵌入知识上下文,从而避免KG中与输入文本不匹配的冗余和模糊知识带来的影响。实验结果表明,Coke在典型的知识驱动NLP任务上优于各种基线方法,表明了利用动态知识上下文进行语言理解的有效性。除了性能提升外,Coke中动态选择的知识能以比传统PLMs更具可解释性的形式描述文本相关知识的语义。我们的源代码和数据集将公开提供以展示Coke的更多细节。

英文摘要:

Several recent efforts have been devoted to enhancing pre-trained language models (PLMs) by utilizing extra heterogeneous knowledge in knowledge graphs (KGs) and achieved consistent improvements on various knowledge-driven NLP tasks. However, most of these knowledge-enhanced PLMs embed static sub-graphs of KGs ("knowledge context"), regardless of that the knowledge required by PLMs may change dynamically according to specific text ("textual context"). In this paper, we propose a novel framework named Coke to dynamically select contextual knowledge and embed knowledge context according to textual context for PLMs, which can avoid the effect of redundant and ambiguous knowledge in KGs that cannot match the input text. Our experimental results show that Coke outperforms various baselines on typical knowledge-driven NLP tasks, indicating the effectiveness of utilizing dynamic knowledge context for language understanding. Besides the performance improvements, the dynamically selected knowledge in Coke can describe the semantics of text-related knowledge in a more interpretable form than the conventional PLMs. Our source code and datasets will be available to provide more details for Coke.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑