Few-shot Learning (FSL) is aimed to make predictions based on a limited number of samples. Structured data such as knowledge graphs and ontology libraries has been leveraged to benefit the few-shot setting in various tasks. However, the priors adopted by the existing methods suffer from challenging knowledge missing, knowledge noise, and knowledge heterogeneity, which hinder the performance for few-shot learning. In this study, we explore knowledge injection for FSL with pre-trained language models and propose ontology-enhanced prompt-tuning (OntoPrompt). Specifically, we develop the ontology transformation based on the external knowledge graph to address the knowledge missing issue, which fulfills and converts structure knowledge to text. We further introduce span-sensitive knowledge injection via a visible matrix to select informative knowledge to handle the knowledge noise issue. To bridge the gap between knowledge and text, we propose a collective training algorithm to optimize representations jointly. We evaluate our proposed OntoPrompt in three tasks, including relation extraction, event extraction, and knowledge graph completion, with eight datasets. Experimental results demonstrate that our approach can obtain better few-shot performance than baselines.
Telefónica R&D（西班牙电信研发部门）；Universidad Politécnica de Madrid（马德里理工大学）
The present paper presents the Weighted Ontology Approximation Heuristic (WOAH), a novel zero-shot approach to ontology estimation for conversational agents development environments. This methodology extracts verbs and nouns separately from data by distilling the dependencies obtained and applying similarity and sparsity metrics to generate an ontology estimation configurable in terms of the level of generalization.
University of Oxford（牛津大学）；INRIA（法国国家信息与自动化研究所）；Zhejiang University（浙江大学）；The University of Edinburgh（爱丁堡大学）；The University of Aberdeen（阿伯丁大学）
Zero-shot learning (ZSL) is a popular research problem that aims at predicting for those classes that have never appeared in the training stage by utilizing the inter-class relationship with some side information. In this study, we propose to model the compositional and expressive semantics of class labels by an OWL (Web Ontology Language) ontology, and further develop a new ZSL framework with ontology embedding. The effectiveness has been verified by some primary experiments on animal image classification and visual question answering.
We propose a novel framework named ViOCE that integrates ontology-based background knowledge in the form of $n$-ball concept embeddings into a neural network based vision architecture. The approach consists of two components - converting symbolic knowledge of an ontology into continuous space by learning n-ball embeddings that capture properties of subsumption and disjointness, and guiding the training and inference of a vision model using the learnt embeddings. We evaluate ViOCE using the task of few-shot image classification, where it demonstrates superior performance on two standard benchma…
Zhejiang University（浙江大学）；University of Oxford（牛津大学）；The University of Edinburgh（爱丁堡大学）；Alibaba Group（阿里巴巴集团）
Knowledge Graph (KG) and its variant of ontology have been widely used for knowledge representation, and have shown to be quite effective in augmenting Zero-shot Learning (ZSL). However, existing ZSL methods that utilize KGs all neglect the intrinsic complexity of inter-class relationships represented in KGs. One typical feature is that a class is often related to other classes in different semantic aspects. In this paper, we focus on ontologies for augmenting ZSL, and propose to learn disentangled ontology embeddings guided by ontology properties to capture and utilize more fine-grained class…
From Prompt to Graph: Comparing LLM-Based Information Extraction Strategies in Domain-Specific Ontology Development
National University of Singapore（新加坡国立大学）；Comptech i Skillingaryd AB（Comptech i Skillingaryd AB公司）；Lappeenranta University of Technology（拉彭兰塔理工大学）；California State University Northridge（加州州立大学北岭分校）
Zhejiang University（浙江大学）；University of Oxford（牛津大学）；University of Edinburgh（爱丁堡大学）；Huawei Technologies Co., Ltd.（华为技术有限公司）
Zero-shot Learning (ZSL), which aims to predict for those classes that have never appeared in the training data, has arisen hot research interests. The key of implementing ZSL is to leverage the prior knowledge of classes which builds the semantic relationship between classes and enables the transfer of the learned models (e.g., features) from training classes (i.e., seen classes) to unseen classes. However, the priors adopted by the existing methods are relatively limited with incomplete semantics. In this paper, we explore richer and more competitive prior knowledge to model the inter-class …
OntoChatGPT Information System: Ontology-Driven Structured Prompts for ChatGPT Meta-Learning
Glushkov Institute of Cybernetics of the National Academy of Sciences of Ukraine（乌克兰国家科学院格鲁什科夫控制论研究所）；Frantsevic Institute for Problems in Material Science of the National Academy of Sciences of Ukraine（乌克兰国家科学院弗兰采维奇材料科学问题研究所）
Ontology Learning with LLMs: A Benchmark Study on Axiom Identification
Netherlands Organisation for Applied Scientific Research (TNO)（荷兰应用科学研究院）；Leiden University Centre for Linguistics (LUCL)（莱顿大学语言研究中心）；Utrecht University（乌得勒支大学）
From Prompts to Context: An Ontology-Driven Framework for Human-Generative AI Collaboration
Gamaizer；Université de technologie de Compiègne, CNRS, Heudiasyc（法国图尔学院、CNRS、Heudiasyc）；Sorbonne Université, CNRS UMR 7585, LPMHE（索邦大学、CNRS UMR 7585、LPMHE）
Heterogeneous LLM Methods for Ontology Learning (Few-Shot Prompting, Ensemble Typing, and Attention-Based Taxonomies)
Skolkovo Institute of Science and Technology（斯克尔科沃科学与技术研究院）；BIMAI-Lab, Biomedically Informed Artificial Intelligence Laboratory, University of Sharjah（沙迦大学生物医学引导人工智能实验室）；Ivannikov Institute for System Programming of the Russian Academy of Sciences（俄罗斯科学院伊万诺夫系统编程研究所）；Lomonosov Moscow State University（莫斯科罗蒙诺索夫莫斯科大学）；AIRI Institute（AIRI研究所）
Leveraging LLMs for Collaborative Ontology Engineering in Parkinson Disease Monitoring and Alerting
Intelligent Systems Laboratory, Department of Cultural Technology and Communication, University of the Aegean（爱琴海大学智能系统实验室）；Artificial Intelligence Laboratory, Department of Digital Systems, University of Piraeus（比雷埃克斯大学人工智能实验室）
Large language models have shown that impressive zero-shot performance can be achieved through natural language prompts (Radford et al., 2019; Brown et al., 2020; Sanh et al., 2021). Creating an effective prompt, however, requires significant trial and error. That \textit{prompts} the question: how do the qualities of a prompt effects its performance? To this end, we collect and standardize prompts from a diverse range of tasks for use with tasks they were not designed for. We then evaluate these prompts across fixed multiple choice datasets for a quantitative analysis of how certain attribute…
New York University（纽约大学）；University of California, San Diego（加州大学圣地亚哥分校）
Prompt-based learning (i.e., prompting) is an emerging paradigm for exploiting knowledge learned by a pretrained language model. In this paper, we propose Automatic Multi-Label Prompting (AMuLaP), a simple yet effective method to automatically select label mappings for few-shot text classification with prompting. Our method exploits one-to-many label mappings and a statistics-based algorithm to select label mappings given a prompt template. Our experiments demonstrate that AMuLaP achieves competitive performance on the GLUE benchmark without human effort or external resources.
Zhejiang University（浙江大学）；Alibaba-Zhejiang University Joint Research Institute of Frontier Technologies（阿里巴巴-浙江大学前沿技术联合研究院）；Alibaba Group（阿里巴巴集团）
Large-scale pre-trained language models have contributed significantly to natural language processing by demonstrating remarkable abilities as few-shot learners. However, their effectiveness depends mainly on scaling the model parameters and prompt design, hindering their implementation in most real-world applications. This study proposes a novel pluggable, extensible, and efficient approach named DifferentiAble pRompT (DART), which can convert small language models into better few-shot learners without any prompt engineering. The main principle behind this approach involves reformulating pote…
Predicting Microbial Ontology and Pathogen Risk from Environmental Metadata with Large Language Models
Department of Electrical and Computer Engineering, Drexel University, Philadelphia, United States（电气与计算机工程系，德雷塞尔大学，费城，美国）；MODULABS, Seoul, Korea（MODULABS，首尔，韩国）
Recent advances in large pre-trained language models (PLMs) lead to impressive gains in natural language understanding (NLU) tasks with task-specific fine-tuning. However, directly fine-tuning PLMs heavily relies on sufficient labeled training instances, which are usually hard to obtain. Prompt-based tuning on PLMs has shown to be powerful for various downstream few-shot tasks. Existing works studying prompt-based tuning for few-shot NLU tasks mainly focus on deriving proper label words with a verbalizer or generating prompt templates to elicit semantics from PLMs. In addition, conventional da…
UC Irvine（加州大学欧文分校）；University of Edinburgh（爱丁堡大学）；UC Berkeley（加州大学伯克利分校）；Facebook AI Research（脸书人工智能研究院）；University College London（伦敦大学学院）
Prompting language models (LMs) with training examples and task descriptions has been seen as critical to recent successes in few-shot learning. In this work, we show that finetuning LMs in the few-shot setting can considerably reduce the need for prompt engineering. In fact, one can use null prompts, prompts that contain neither task-specific templates nor training examples, and achieve competitive accuracy to manually-tuned prompts across a wide range of tasks. While finetuning LMs does introduce new parameters for each downstream task, we show that this memory overhead can be substantially …
Metacognition-Enhanced Few-Shot Prompting With Positive Reinforcement
Institute of AI for Education, East China Normal University（华东师范大学教育人工智能研究院）；School of Computer Science and Technology, East China Normal University（华东师范大学计算机科学与技术学院）；Shanghai Changning Mental Health Center（上海市长宁区精神卫生中心）
University of Chinese Academy of Sciences（中国科学院大学）；Institute of Information Engineering（信息工程研究所）；Beijing Institute of Technology（北京理工大学）；Guangzhou University（广州大学）
The effectiveness of prompt learning has been demonstrated in different pre-trained language models. By formulating suitable template and choosing representative label mapping, prompt learning can be used as an efficient knowledge probe. However, finding suitable prompt in existing methods requires multiple experimental attempts or appropriate vector initialization on formulating suitable template and choosing representative label mapping, which it is more common in few-shot learning tasks. Motivating by PLM working process, we try to construct the prompt from task semantic perspective and thu…
Cell-ontology guided transcriptome foundation model
Mila - Québec AI Institute（米拉-魁北克人工智能研究所）；University of Montréal（蒙特利尔大学）；McGill University（麦吉尔大学）；Cornell University（康奈尔大学）；HEC Montréal（蒙特利尔高等商学院）；CIFAR AI Chair（加拿大高级研究所人工智能主席职位）
Institute for Clarity in Documentation（文档清晰性研究所）；Inria Paris-Rocquencourt（法国国家信息与自动化研究所巴黎-罗克库尔中心）；Rajiv Gandhi University（拉吉夫·甘地大学）；Tsinghua University（清华大学）；Palmer Research Laboratories（帕尔默研究实验室）
Prevailing methods for mapping large generative language models to supervised tasks may fail to sufficiently probe models' novel capabilities. Using GPT-3 as a case study, we show that 0-shot prompts can significantly outperform few-shot prompts. We suggest that the function of few-shot examples in these cases is better described as locating an already learned task rather than meta-learning. This analysis motivates rethinking the role of prompts in controlling and evaluating powerful language models. In this work, we discuss methods of prompt programming, emphasizing the usefulness of consider…
Pre-trained masked language models successfully perform few-shot learning by formulating downstream tasks as text infilling. However, as a strong alternative in full-shot settings, discriminative pre-trained models like ELECTRA do not fit into the paradigm. In this work, we adapt prompt-based few-shot learning to ELECTRA and show that it outperforms masked language models in a wide range of tasks. ELECTRA is pre-trained to distinguish if a token is generated or original. We naturally extend that to prompt-based few-shot learning by training to score the originality of the target options withou…
