arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1909.07606cs.CLcs.LG
收藏

K-BERT:用知识图谱实现语言表示

K-BERT: Enabling Language Representation with Knowledge Graph

Peking University(北京大学);Tencent Research(腾讯研究院);Beijing Normal University(北京师范大学)

Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, Ping Wang

摘要:

预先训练的语言表示模型,如BERT,从大规模语料库中捕获通用语言表示,但缺乏特定领域的知识。在阅读领域文本时,专家会利用相关知识进行推理。为了使机器实现这种能力,我们提出了一个知识使能的语言表示模型(K-BERT)与知识图谱(KG),其中三元组被注入到句子中作为领域知识。然而,过多的知识融入可能会偏离句子的正确含义,这就是知识噪音(KN)问题。为了克服知识噪声,K-BERT引入软位置和可见矩阵来限制知识的影响。K-BERT可以通过配备KG轻松地将领域知识注入到模型中,而无需自行进行预训练,因为它能够从预训练的BERT中加载模型参数。我们的调查显示,在12个NLP任务的可喜成果。特别是在特定领域的任务(包括金融,法律和医学),K-BERT显着优于BERT,这表明K-BERT是一个很好的选择,解决知识驱动的问题,需要专家。

英文摘要:

Pre-trained language representation models, such as BERT, capture a general language representation from large-scale corpora, but lack domain-specific knowledge. When reading a domain text, experts make inferences with relevant knowledge. For machines to achieve this capability, we propose a knowledge-enabled language representation model (K-BERT) with knowledge graphs (KGs), in which triples are injected into the sentences as domain knowledge. However, too much knowledge incorporation may divert the sentence from its correct meaning, which is called knowledge noise (KN) issue. To overcome KN, K-BERT introduces soft-position and visible matrix to limit the impact of knowledge. K-BERT can easily inject domain knowledge into the models by equipped with a KG without pre-training by-self because it is capable of loading model parameters from the pre-trained BERT. Our investigation reveals promising results in twelve NLP tasks. Especially in domain-specific tasks (including finance, law, and medicine), K-BERT significantly outperforms BERT, which demonstrates that K-BERT is an excellent choice for solving the knowledge-driven problems that require experts.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑