In this work, we aim at equipping pre-trained language models with structured knowledge. We present two self-supervised tasks learning over raw text with the guidance from knowledge graphs. Building upon entity-level masked language models, our first contribution is an entity masking scheme that exploits relational knowledge underlying the text. This is fulfilled by using a linked knowledge graph to select informative entities and then masking their mentions. In addition we use knowledge graphs to obtain distractors for the masked entities, and propose a novel distractor-suppressed ranking objective which is optimized jointly with masked language model. In contrast to existing paradigms, our approach uses knowledge graphs implicitly, only during pre-training, to inject language models with structured knowledge via learning from raw text. It is more efficient than retrieval-based methods that perform entity linking and integration during finetuning and inference, and generalizes more effectively than the methods that directly learn from concatenated graph triples. Experiments show that our proposed model achieves improved performance on five benchmark datasets, including question answering and knowledge base completion tasks.
PCA-Guided Autoencoding for Structured Dimensionality Reduction in Active Infrared Thermography
Khalifa University of Science and Technology（卡利法科技科学大学）；Advanced Research and Innovation Center（先进研究与创新中心）；Department of Aerospace Engineering（航空航天工程系）；BioMedIA Lab（BioMedIA实验室）；Mohamed Bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Department of Electrical Engineering and Computer Science（电气工程与计算机科学系）；Department of Information Systems and Operations Research（信息系统与运营管理系）；Vienna University of Economics and Business（维也纳经济与商业大学）；Department of Industrial and Information Engineering and Economics (DIIIE)（工业与信息工程与经济学系）；University of L’Aquila（拉奎拉大学）；Department of Industrial and Systems Engineering（工业与系统工程系）
SCOPE: Structured Prototype-Guided Adaptation for EEG Foundation Models with Limited Labels
Saw Swee Hock School of Public Health, National University of Singapore（新加坡国立大学 Saw Swee Hock 公共卫生学院）；Institute of Data Science, National University of Singapore（新加坡国立大学数据科学研究所）；Guangzhou Research Translation and Innovation Institute, National University of Singapore（新加坡国立大学广州研究翻译与创新研究所）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）；Beijing Key Laboratory of Brainnetome and Brain-Computer Interface, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所脑网络与脑机接口重点实验室）；Brainnetome Center, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所脑网络中心）
Singapore University of Technology and Design（新加坡科技设计大学）
Named entity recognition (NER), which focuses on the extraction of semantically meaningful named entities and their semantic classes from text, serves as an indispensable component for several down-stream natural language processing (NLP) tasks such as relation extraction and event extraction. Dependency trees, on the other hand, also convey crucial semantic-level information. It has been shown previously that such information can be used to improve the performance of NER (Sasano and Kurohashi 2008, Ling and Weld 2012). In this work, we investigate on how to better utilize the structured infor…
Extracting temporal relations (e.g., before, after, and simultaneous) among events is crucial to natural language understanding. One of the key challenges of this problem is that when the events of interest are far away in text, the context in-between often becomes complicated, making it challenging to resolve the temporal relationship between them. This paper thus proposes a new Syntax-guided Graph Transformer network (SGT) to mitigate this issue, by (1) explicitly exploiting the connection between two events based on their dependency parsing trees, and (2) automatically locating temporal cue…
TemplateRL: Structured Template-Guided Reinforcement Learning for LLM Reasoning
Tsinghua University（清华大学）；Nanyang Technological University（南洋理工大学）；Princeton University（普林斯顿大学）；Shanghai AI Lab（上海人工智能实验室）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）
GRASP: Plan-Guided Graph Retrieval with Adaptive Fusion and Reranking on Semi-Structured Knowledge Bases
Department of Electrical Engineering and Computer Science, University of Michigan（密歇根大学电气工程与计算机科学系）；Department of Computational Medicine and Bioinformatics, University of Michigan（密歇根大学计算医学与生物信息学系）
Dependency-Guided Code Generation: Structured Matrix Decomposition and Consistency-Guided Refinement
University of the Chinese Academy of Sciences（中国科学院大学）；ByteDance Inc.（字节跳动公司）；Zhejiang University（浙江大学）；National Taiwan University（台湾国立大学）；Peking University（北京大学）；Alibaba Group（阿里巴巴集团）；Tsinghua University（清华大学）；Liaoning Technical University（辽宁技术大学）
Institute of Informatics & Telecommunications, NCSR “Demokritos”（德谟克利特国家科学研究中心信息与电信研究所）；Department of Informatics, Athens University of Economics and Business（雅典经济与商业大学信息学系）
Although BERT is widely used by the NLP community, little is known about its inner workings. Several attempts have been made to shed light on certain aspects of BERT, often with contradicting conclusions. A much raised concern focuses on BERT's over-parameterization and under-utilization issues. To this end, we propose o novel approach to fine-tune BERT in a structured manner. Specifically, we focus on Large Scale Multilabel Text Classification (LMTC) where documents are assigned with one or more labels from a large predefined set of hierarchically organized labels. Our approach guides specifi…
Sun Yat-Sen University（中山大学）；Microsoft Research Asia（微软亚洲研究院）
Recent neural models for data-to-text generation are mostly based on data-driven end-to-end training over encoder-decoder networks. Even though the generated texts are mostly fluent and informative, they often generate descriptions that are not consistent with the input structured data. This is a critical issue especially in domains that require inference or calculations over raw data. In this paper, we attempt to improve the fidelity of neural data-to-text generation by utilizing pre-executed symbolic operations. We propose a framework called Operation-guided Attention-based sequence-to-seque…
In recent years, transformer-based language models have achieved state of the art performance in various NLP benchmarks. These models are able to extract mostly distributional information with some semantics from unstructured text, however it has proven challenging to integrate structured information, such as knowledge graphs into these models. We examine a variety of approaches to integrate structured knowledge into current language models and determine challenges, and possible opportunities to leverage both structured and unstructured information sources. From our survey, we find that there …
We propose a structured extension to bidirectional-context conditional language generation, or "infilling," inspired by Frame Semantic theory (Fillmore, 1976). Guidance is provided through two approaches: (1) model fine-tuning, conditioning directly on observed symbolic frames, and (2) a novel extension to disjunctive lexically constrained decoding that leverages frame semantic lexical units. Automatic and human evaluations confirm that frame-guided generation allows for explicit manipulation of intended infill semantics, with minimal loss in distinguishability from human-generated text. Our m…
Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）；Center for Data Science, Peking University（北京大学数据科学中心）；The MOE Key Laboratory of Computational Linguistics, Peking University（北京大学计算语言学教育部重点实验室）
Neural text generation models are likely to suffer from the low-diversity problem. Various decoding strategies and training-based methods have been proposed to promote diversity only by exploiting contextual features, but rarely do they consider incorporating syntactic structure clues. In this work, we propose using linguistic annotation, i.e., part-of-speech (POS), to guide the text generation. In detail, we introduce POS Guided Softmax to explicitly model two posterior probabilities: (i) next-POS, and (ii) next-token from the vocabulary of the target POS. A POS Guided Sampling strategy is fu…
InsightGUIDE: An Opinionated AI Assistant for Guided Critical Reading of Scientific Literature
Information Management Systems Institute（信息管理系统研究所）；ATHENA Research Centre（ATHENA研究中心）；Department of Informatics and Telecommunications（信息与电信系）；University of the Peloponnese（希腊皮埃里亚大学）
