Prompt tuning is a parameter-efficient approach to adapting pre-trained language models to downstream tasks. Although prompt tuning has been shown to match the performance of full model tuning when training data is sufficient, it tends to struggle in few-shot learning settings. In this paper, we present Multi-task Pre-trained Modular Prompt (MP2) to boost prompt tuning for few-shot learning. MP2 is a set of combinable prompts pre-trained on 38 Chinese tasks. On downstream tasks, the pre-trained prompts are selectively activated and combined, leading to strong compositional generalization to unseen tasks. To bridge the gap between pre-training and fine-tuning, we formulate upstream and downstream tasks into a unified machine reading comprehension task. Extensive experiments under two learning paradigms, i.e., gradient descent and black-box tuning, show that MP2 significantly outperforms prompt tuning, full model tuning, and prior prompt pre-training methods in few-shot settings. In addition, we demonstrate that MP2 can achieve surprisingly fast and strong adaptation to downstream tasks by merely learning 8 parameters to combine the pre-trained modular prompts.
A Modular Multitask Reasoning Framework Integrating Spatio-temporal Models and LLMs
College of Computing and Data Science, Nanyang Technological University, Singapore（南洋理工大学计算机与数据科学学院）；Institute for Infocomm Research, A*STAR, Singapore（资讯通信研究院）；School of Computer Science and Engineering, Beihang University, China（北京航空航天大学计算机科学与工程学院）
Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging
School of Computer Science & Technology, Huazhong University of Science and Technology（华中科技大学计算机科学与技术学院）；Joint Laboratory of HUST and Pingan Property & Casualty Research (HPL)（华中科技大学与平安产险联合实验室）；Ping An Property & Casualty Insurance Company of China, Ltd.（中国平安财产保险股份有限公司）；The Chinese University of Hong Kong（香港中文大学）
