持续学习(Continual Learning,CL)近年来受到越来越多的关注,其目标是模拟人类在学习新概念时不发生灾难性遗忘的能力。现有持续学习方法虽在一定程度上实现了这一目标,但仍容易出现已学习特征空间的语义漂移问题。从超大规模数据集中习得、具备鲁棒特征表示能力的基础模型,为解决持续学习问题提供了理想的基础。近期研究也表明,可通过提示调优技术让基础模型适配特定任务,且该过程基本不会破坏特征表示的通用性。然而,一个尚未解决的问题是如何同时学习任务专属提示与全局提示(即捕捉跨任务信息的提示)。在本研究中,我们提出Prompt Of Prompts(POP,提示之提示)模型来解决这一问题:该模型逐步学习一组任务专属提示,以及一组被称为POP的全局提示,用以整合来自任务专属提示的信息。我们的实验表明,搭载POP学习机制的基础模型,性能能够大幅超越经典持续学习方法。此外,由于提示调优仅需少量训练样本,POP可在少样本设置下执行持续学习,且性能仍优于在完整数据集上训练的同类竞争方法。
英文摘要:
Continual learning (CL) has attracted increasing attention in the recent past. It aims to mimic the human ability to learn new concepts without catastrophic forgetting. While existing CL methods accomplish this to some extent, they are still prone to semantic drift of the learned feature space. Foundation models, which are endowed with a robust feature representation, learned from very large datasets, provide an interesting substrate for the solution of the CL problem. Recent work has also shown that they can be adapted to specific tasks by prompt tuning techniques that leave the generality of the representation mostly unscathed. An open question is, however, how to learn both prompts that are task specific and prompts that are global, i.e. capture cross-task information. In this work, we propose the Prompt Of Prompts (POP) model, which addresses this goal by progressively learning a group of task-specified prompts and a group of global prompts, denoted as POP, to integrate information from the former. We show that a foundation model equipped with POP learning is able to outperform classic CL methods by a significant margin. Moreover, as prompt tuning only requires a small set of training samples, POP is able to perform CL in the few-shot setting, while still outperforming competing methods trained on the entire dataset.
Pengcheng laboratory(鹏城实验室);Harbin Institute of Technology(哈尔滨工业大学);Xi’an Jiaotong University(西安交通大学);Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院)
Consistent Prompting for Rehearsal-Free Continual Learning
School of Artificial Intelligence, Sun Yat-sen University(中山大学人工智能学院);Cheng Kar-Shun Robotics Institute, The Hong Kong University of Science and Technology(香港科技大学郑家纯机器人研究院);Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education(教育部机器智能与先进计算重点实验室)
MM-Prompt: Cross-Modal Prompt Tuning for Continual Visual Question Answering
Khoury College of Computer Sciences, Northeastern University(东北大学克劳尔计算机科学学院);New Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所模式识别新实验室)
Adapter-Enhanced Semantic Prompting for Continual Learning
Beijing Key Laboratory of Multimedia and Intelligent Software Technology(北京多媒体与智能软件技术重点实验室);Faculty of Information Technology, Beijing University of Technology(信息科技学院,北京理工大学);University of California at Merced(加州大学默塞德分校);Macquarie University(麦考瑞大学)
Dynamic Cross-Modal Prompt Generation for Multimodal Continual Instruction Tuning
School of Artificial Intelligence, Nanjing University(南京大学人工智能学院);State Key Laboratory for Novel Software Technology, Nanjing University(南京大学新型软件技术国家重点实验室)
C^2Prompt: Class-aware Client Knowledge Interaction for Federated Continual Learning
Wangxuan Institute of Computer Technology Peking University(北京大学王轩计算机技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Inner Mongolia University of Technology(内蒙古科技大学)
Multiple Queries with Multiple Keys: A Precise Prompt Matching Paradigm for Prompt-based Continual Learning
National Key Laboratory for Novel Software Technology, Nanjing University(新型软件技术国家重点实验室,南京大学);School of Electronic Science and Engineering, Nanjing University(电子科学与工程学院,南京大学)
Critical Patch-Aware Sparse Prompting with Decoupled Training for Continual Learning on the Edge
School of Computer Science and Engineering, Chung-Ang University(中央大学计算机科学与工程学院);Department of Artificial Intelligence, Chung-Ang University(中央大学人工智能系)
Dynamically Anchored Prompting for Task-Imbalanced Continual Learning
Xiamen University(厦门大学);Univ. Grenoble Alpes(格勒诺布尔大学);Inria(法国国家信息与自动化研究所);CNRS(法国国家科学研究中心);Grenoble INP(格勒诺布尔理工学院);LJK(让·昆廷诺实验室);Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)(广东省人工智能与数字经济实验室(深圳))
PromptFusion: Decoupling Stability and Plasticity for Continual Learning
Shanghai Collaborative Innovation Center of Intelligent Visual Computing(上海智能视觉计算协同创新中心);Shanghai Key Lab of Intell. Info. Processing, School of CS, Fudan University(复旦大学计算机科学学院上海市智能信息处理重点实验室);Huya Inc(虎牙公司);Cornell Univeristy(康奈尔大学)
CM2-Net: Continual Cross-Modal Mapping Network for Driver Action Recognition
School of Electrical and Electronic Engineering, Nanyang Technological University(南洋理工大学电气与电子工程学院);Continental-NTU Corporate Lab, Nanyang Technological University(大陆集团-南洋理工大学联合企业实验室);College of Computer Science and Technology, Harbin Engineering University(哈尔滨工程大学计算机科学与技术学院)
GAP-Prompt: Gated Adaptive Prompting for Efficient Continual Learning
Université d’Orléans(奥尔良大学);INSA CVL(中央卢瓦尔河谷国立应用科学学院);CY Cergy Paris University(塞尔吉-蓬图瓦兹大学);ENSEA(法国高等电子与数字学院);CNRS(法国国家科学研究中心);Université de Technologie de Troyes(特鲁瓦技术大学)
Hierarchical Prompts for Rehearsal-free Continual Learning
University of Science and Technology of China(中国科学技术大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Tianjin University of Technology(天津理工大学);University of Chinese Academy of Sciences(中国科学院大学)
FPPL: An Efficient and Non-IID Robust Federated Continual Learning Framework
School of Computer Science and Technology, East China Normal University(华东师范大学计算机科学与技术学院);School of Software Engineering, TongJi University(同济大学软件工程学院)
PROL : Rehearsal Free Continual Learning in Streaming Data via Prompt Online Learning
STEM University of South Australia(南澳大利亚STEM大学);Institute for Infocomm Research, A*STAR & IPAL(信息与通信研究所,A*STAR及IPAL);Systems Research Institute, Polish Academy of Sciences(波兰科学院系统研究所)
GRID: Scaling Task-Agnostic Inference in Continual Prompt Tuning
State University of New York at Buffalo(纽约州立大学布法罗分校);Department of Computer Science and Engineering(计算机科学与工程系);Institute for Artificial Intelligence and Data Science(人工智能与数据科学研究院)
From a Prompt to Repertoires: Evolving Functional REpertoires Enable LLM Continual Learning
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Shanghai AI Laboratory(上海人工智能实验室);University of Science and Technology of China(中国科学技术大学);New Jersey Institute of Technology(新泽西理工学院)