Multi-task Pre-training Language Model for Semantic Network Completion
Tencent（腾讯）；Beijing Institute of Microbiology and Epidemiology（北京微生物与流行病学研究所）；State Key Laboratory of Pathogen and Biosecurity（病原与生物安全国家重点实验室）；National University of Defense Technology（国防科技大学）；National Key Lab of Parallel and Distributed Processing（并行与分布式处理国家重点实验室）
Da Li, Sen Yang, Kele Xu, Ming Yi, Yukai He, Huaimin Wang
Semantic networks, such as the knowledge graph, can represent the knowledge leveraging the graph structure. Although the knowledge graph shows promising values in natural language processing, it suffers from incompleteness. This paper focuses on knowledge graph completion by predicting linkage between entities, which is a fundamental yet critical task. Semantic matching is a potential solution as it can deal with unseen entities, which the translational distance based methods struggle with. However, to achieve competitive performance as translational distance based methods, semantic matching based methods require large-scale datasets for the training purpose, which are typically unavailable in practical settings. Therefore, we employ the language model and introduce a novel knowledge graph architecture named LP-BERT, which contains two main stages: multi-task pre-training and knowledge graph fine-tuning. In the pre-training phase, three tasks are taken to drive the model to learn the relationship from triples by predicting either entities or relations. While in the fine-tuning phase, inspired by contrastive learning, we design a triple-style negative sampling in a batch, which greatly increases the proportion of negative sampling while keeping the training time almost unchanged. Furthermore, we propose a new data augmentation method utilizing the inverse relationship of triples to improve the performance and robustness of the model. To demonstrate the effectiveness of our method, we conduct extensive experiments on three widely-used datasets, WN18RR, FB15k-237, and UMLS. The experimental results demonstrate the superiority of our methods, and our approach achieves state-of-the-art results on WN18RR and FB15k-237 datasets. Significantly, Hits@10 indicator is improved by 5% from previous state-of-the-art result on the WN18RR dataset while reaching 100% on the UMLS dataset.
Peking University（北京大学）；Washington University in St. Louis（圣路易斯华盛顿大学）；UC Berkeley（加州大学伯克利分校）
The task of completing knowledge triplets has broad downstream applications. Both structural and semantic information plays an important role in knowledge graph completion. Unlike previous approaches that rely on either the structures or semantics of the knowledge graphs, we propose to jointly embed the semantics in the natural language description of the knowledge triplets with their structure information. Our method embeds knowledge graphs for the completion task via fine-tuning pre-trained language models with respect to a probabilistic structured loss, where the forward pass of the languag…
Allen Institute for Artificial Intelligence（艾伦人工智能研究所）；University of Washington（华盛顿大学）
Automatic KB completion for commonsense knowledge graphs (e.g., ATOMIC and ConceptNet) poses unique challenges compared to the much studied conventional knowledge bases (e.g., Freebase). Commonsense knowledge graphs use free-form text to represent nodes, resulting in orders of magnitude more nodes compared to conventional KBs (18x more nodes in ATOMIC compared to Freebase (FB15K-237)). Importantly, this implies significantly sparser graph structures - a major challenge for existing KB completion methods that assume densely connected graphs over a relatively smaller set of nodes. In this paper,…
The fixpoint completion fix(P) of a normal logic program P is a program transformation such that the stable models of P are exactly the models of the Clark completion of fix(P). This is well-known and was studied by Dung and Kanchanasut (1989). The correspondence, however, goes much further: The Gelfond-Lifschitz operator of P coincides with the immediate consequence operator of fix(P), as shown by Wendt (2002), and even carries over to standard operators used for characterizing the well-founded and the Kripke-Kleene semantics. We will apply this knowledge to the study of the stable semantics,…
Flow-Modulated Scoring for Semantic-Aware Knowledge Graph Completion
Dalian University of Technology（大连理工大学）；Beijing Institute of Technology（北京理工大学）；University of Manchester（曼彻斯特大学）；Renmin University of China（中国人民大学）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Beijing Unisound Information Technology Co., Ltd（北京云知声信息技术有限公司）
Background Knowledge graphs (KGs), especially medical knowledge graphs, are often significantly incomplete, so it necessitating a demand for medical knowledge graph completion (MedKGC). MedKGC can find new facts based on the exited knowledge in the KGs. The path-based knowledge reasoning algorithm is one of the most important approaches to this task. This type of method has received great attention in recent years because of its high performance and interpretability. In fact, traditional methods such as path ranking algorithm (PRA) take the paths between an entity pair as atomic features. Howe…
The Weak Completion Semantics (WCS) is a computational cognitive theory that has shown to be successful in modeling episodes of human reasoning. As the WCS is a recently developed logic programming approach, this paper investigates the correspondence of the WCS with respect to the well-established Answer Set Semantics (ASP). The underlying three-valued logic of both semantics is different and their models are evaluated with respect to different program transformations. We first illustrate these differences by the formal representation of some examples of a well-known psychological experiment, …
Cumulative Path-Level Semantic Reasoning for Inductive Knowledge Graph Completion
Nanjing University of Science and Technology（南京理工大学）；Jinan University（济南大学）；Guilin University of Electronic Technology（桂林电子科技大学）；Shanxi University（山西大学）；Nanyang Technological University（南洋理工大学）；Beijing University Of Technology（北京理工大学）
S$^2$DN: Learning to Denoise Unconvincing Knowledge for Inductive Knowledge Graph Completion
Hunan University（湖南大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Xiangtan University（湘潭大学）
We investigate the relationship between the generalization of program completion defined in 1984 by Lloyd and Topor and the generalization of the stable model semantics introduced recently by Ferraris et al. The main theorem can be used to characterize, in some cases, the general stable models of a logic program by a first-order formula. The proof uses Truszczynski's stable model semantics of infinitary propositional formulas.
Context-Driven Knowledge Graph Completion with Semantic-Aware Relational Message Passing
School of Computer Science and Technology, Dalian University of Technology, Dalian, Liaoning, China（计算机科学与技术学院，大连理工大学，大连，辽宁，中国）；Beijing Institute of Technology, Beijing, China（北京理工大学，北京，中国）；School of Mathematical Sciences, Dalian University of Technology, Dalian, Liaoning, China（数学学院，大连理工大学，大连，辽宁，中国）；School of Control Science and Engineering, Dalian University of Technology, Dalian, Liaoning, China（控制科学与工程学院，大连理工大学，大连，辽宁，中国）
Institute of Information Engineering, CAS（中国科学院信息工程研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Institute of Computing Technology, CAS（中国科学院计算技术研究所）；Peng Cheng Laboratory（鹏城实验室）
Tensor factorization and distanced based models play important roles in knowledge graph completion (KGC). However, the relational matrices in KGC methods often induce a high model complexity, bearing a high risk of overfitting. As a remedy, researchers propose a variety of different regularizers such as the tensor nuclear norm regularizer. Our motivation is based on the observation that the previous work only focuses on the "size" of the parametric space, while leaving the implicit semantic information widely untouched. To address this issue, we propose a new regularizer, namely, Equivariance …
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Carnegie Mellon University（卡内基梅隆大学）；University of Washington（华盛顿大学）
Taxonomies are fundamental to many real-world applications in various domains, serving as structural representations of knowledge. To deal with the increasing volume of new concepts needed to be organized as taxonomies, researchers turn to automatically completion of an existing taxonomy with new concepts. In this paper, we propose TaxoEnrich, a new taxonomy completion framework, which effectively leverages both semantic features and structural information in the existing taxonomy and offers a better representation of candidate position to boost the performance of taxonomy completion. Specific…
College of Computer Science and Information Engineering, Harbin Normal University（哈尔滨师范大学计算机科学与信息工程学院）
Translation models tend to ignore the rich semantic information in triads in the process of knowledge graph complementation. To remedy this shortcoming, this paper constructs a knowledge graph complementation method that incorporates adaptively enhanced semantic information. The hidden semantic information inherent in the triad is obtained by fine-tuning the BERT model, and the attention feature embedding method is used to calculate the semantic attention scores between relations and entities in positive and negative triads and incorporate them into the structural information to form a soft co…
Writing tests is a time-consuming yet essential task during software development. We propose to leverage recent advances in deep learning for text and code generation to assist developers in writing tests. We formalize the novel task of test completion to automatically complete the next statement in a test method based on the context of prior statements and the code under test. We develop TeCo -- a deep learning model using code semantics for test completion. The key insight underlying TeCo is that predicting the next statement in a test method requires reasoning about code execution, which is…
College of Computer Science\ , TMCC, TBI Center\ University Tianjin China；School of Software\ University Tianjin China；College of Computer Science\ , TMCC, TBI Center\ University；School of Software\ University
Cross-modal alignment is essential for vision-language pre-training (VLP) models to learn the correct corresponding information across different modalities. For this purpose, inspired by the success of masked language modeling (MLM) tasks in the NLP pre-training area, numerous masked modeling tasks have been proposed for VLP to further promote cross-modal interactions. The core idea of previous masked modeling tasks is to focus on reconstructing the masked tokens based on visible context for learning local-to-local alignment. However, most of them pay little attention to the global semantic fe…
Washington University in St. Louis（圣路易斯华盛顿大学）；The Ohio State University（俄亥俄州立大学）；National Institutes of Health（美国国立卫生研究院）
Analogy completion has been a popular task in recent years for evaluating the semantic properties of word embeddings, but the standard methodology makes a number of assumptions about analogies that do not always hold, either in recent benchmark datasets or when expanding into other domains. Through an analysis of analogies in the biomedical domain, we identify three assumptions: that of a Single Answer for any given analogy, that the pairs involved describe the Same Relationship, and that each pair is Informative with respect to the other. We propose modifying the standard methodology to relax…
