Prompt-based learning methods in semi-supervised learning (SSL) settings have been shown to be effective on multiple natural language understanding (NLU) datasets and tasks in the literature. However, manually designing multiple prompts and verbalizers requires domain knowledge and human effort, making it difficult and expensive to scale across different datasets. In this paper, we propose two methods to automatically design multiple prompts and integrate automatic verbalizer in SSL settings without sacrificing performance. The first method uses various demonstration examples with learnable continuous prompt tokens to create diverse prompt models. The second method uses a varying number of soft prompt tokens to encourage language models to learn different prompts. For the verbalizer, we use the prototypical verbalizer to replace the manual one. In summary, we obtained the best average accuracy of 73.2% (a relative improvement of 2.52% over even the previous state-of-the-art SSL method with manual prompts and verbalizers) in different few-shot learning settings.
Tsinghua University（清华大学）；Alibaba Group（阿里巴巴集团）；Seoul National University（首尔国立大学）
As an effective approach to tune pre-trained language models (PLMs) for specific tasks, prompt-learning has recently attracted much attention from researchers. By using \textit{cloze}-style language prompts to stimulate the versatile knowledge of PLMs, prompt-learning can achieve promising results on a series of NLP tasks, such as natural language inference, sentiment classification, and knowledge probing. In this work, we investigate the application of prompt-learning on fine-grained entity typing in fully supervised, few-shot and zero-shot scenarios. We first develop a simple and effective p…
Prompt-based Pseudo-labeling Strategy for Sample-Efficient Semi-Supervised Extractive Summarization
David R. Cheriton School of Computer Science（大卫·R·切里顿计算机科学学院）；University of Waterloo（滑铁卢大学）；ServiceNow Research（ServiceNow研究院）；University of British Columbia（不列颠哥伦比亚大学）
ProMedTS: A Self-Supervised, Prompt-Guided Multimodal Approach for Integrating Medical Text and Time Series
Hong Kong Baptist University（香港 Baptist 大学）；Shanxi University（山西大学）；Shanghai Institute for Advanced Study of Zhejiang University（浙江大学上海研究院）；Tencent AI Lab（腾讯人工智能实验室）；Manchester Metropolitan University（曼彻斯特 Metropolitan 大学）；The University of Manchester（曼彻斯特大学）
Weakly-supervised learning (WSL) has shown promising results in addressing label scarcity on many NLP tasks, but manually designing a comprehensive, high-quality labeling rule set is tedious and difficult. We study interactive weakly-supervised learning -- the problem of iteratively and automatically discovering novel labeling rules from data to improve the WSL model. Our proposed model, named PRBoost, achieves this goal via iterative prompt-based rule discovery and model boosting. It uses boosting to identify large-error instances and then discovers candidate rules from them by prompting pre-…
MoPE: Mixture of Prompt Experts for Parameter-Efficient and Scalable Multimodal Fusion
Department of Computing, The Hong Kong Polytechnic University（计算系，香港理工大学）；Research Institute of Multiple Agents and Embodied Intelligence, Pengcheng Laboratory（多智能体与具身智能研究院，鹏城实验室）
We demonstrate that co-training (Blum & Mitchell, 1998) can improve the performance of prompt-based learning by using unlabeled data. While prompting has emerged as a promising paradigm for few-shot and zero-shot learning, it is often brittle and requires much larger models compared to the standard supervised setup. We find that co-training makes it possible to improve the original prompt model and at the same time learn a smaller, downstream task-specific model. In the case where we only have partial access to a prompt model (e.g., output probabilities from GPT-3 (Brown et al., 2020)) we lear…
We present a method for transferring pre-trained self-supervised (SSL) speech representations to multiple languages. There is an abundance of unannotated speech, so creating self-supervised representations from raw audio and fine-tuning on small annotated datasets is a promising direction to build speech recognition systems. SSL models generally perform SSL on raw audio in a pre-training phase and then fine-tune on a small fraction of annotated data. Such models have produced state of the art results for ASR. However, these models are very expensive to pre-train. We use an existing wav2vec 2.0…
German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）
Prompting pre-trained language models has achieved impressive performance on various NLP tasks, especially in low data regimes. Despite the success of prompting in monolingual settings, applying prompt-based methods in multilingual scenarios has been limited to a narrow set of tasks, due to the high cost of handcrafting multilingual prompts. In this paper, we present the first work on prompt-based multilingual relation classification (RC), by introducing an efficient and effective method that constructs prompts from relation triples and involves only minimal translation for the class labels. W…
Beyond Retrieval: Learning Compact User Representations for Scalable LLM Personalization
Microsoft（微软公司）；Shanghai International Studies University（上海国际问题研究大学）；Zhejiang University（浙江大学）；Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University（数据科学与人工智能系，香港理工大学）
