LW2G: Learning Whether to Grow for Prompt-based Continual Learning
College of Computer Science and Technology, Zhejiang University, China(浙江大学计算机科学与技术学院);State Key Laboratory for Novel Software Technology, Nanjing University, China(南京大学新型软件技术国家重点实验室);Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi(阿布扎赫德莫伊丁·本·扎耶德人工智能大学);Huawei Zurich Research Center, Switzerland(华为苏黎世研究中心)
Recent Prompt-based Continual learning (PCL) has achieved remarkable performance with pre-trained models. These approaches expand a prompt pool by adding a new set of prompts while learning and select the correct set during inference. Previous studies have revealed that learning task-wised prompt sets individually and low selection accuracy pose challenges to the performance of PCL. In this paper, we propose a plug-in method, $\textbf{L}$earning $\textbf{W}$hether $\textbf{t}$o $\textbf{G}$row $\textbf{(LW2G)}$, which leverages the disparities between tasks to form an effective and efficient prompt sets pool, thereby achieving intra-task knowledge sharing and cooperation and avoiding the unbounded increase in the cost of the prompt pool. Specifically, a shared set is utilized when several tasks share certain commonalities, and a new set is added when there are significant differences between the new and previous tasks. To achieve this, we develop a metric called Hinder Forward Capability (HFC) to measure the hindrance imposed on learning new tasks by surgically modifying the original gradient onto the orthogonal complement of the old feature space. With HFC, an automated scheme, Dynamic Growing Approach, adaptively learns whether to grow with a dynamic threshold. Furthermore, we design a gradient-based constraint to ensure consistency between the updating prompts and pre-trained knowledge. Extensive experiments show the effectiveness of our method. Code is available at https://github.com/RAIAN08/LW2G.
Multiple Queries with Multiple Keys: A Precise Prompt Matching Paradigm for Prompt-based Continual Learning
National Key Laboratory for Novel Software Technology, Nanjing University(新型软件技术国家重点实验室,南京大学);School of Electronic Science and Engineering, Nanjing University(电子科学与工程学院,南京大学)
PromptFusion: Decoupling Stability and Plasticity for Continual Learning
Shanghai Collaborative Innovation Center of Intelligent Visual Computing(上海智能视觉计算协同创新中心);Shanghai Key Lab of Intell. Info. Processing, School of CS, Fudan University(复旦大学计算机科学学院上海市智能信息处理重点实验室);Huya Inc(虎牙公司);Cornell Univeristy(康奈尔大学)
Hierarchical Prompts for Rehearsal-free Continual Learning
University of Science and Technology of China(中国科学技术大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Tianjin University of Technology(天津理工大学);University of Chinese Academy of Sciences(中国科学院大学)
MergeSlide: Continual Model Merging and Task-to-Class Prompt-Aligned Inference for Lifelong Learning on Whole Slide Images
Nara Institute of Science and Technology(奈良科学技术研究所);Graduate School of Data Science, Chonnam National University(全南国立大学数据科学研究生院);University of Information Technology, Viet Nam National University Ho Chi Minh City(越南胡志明市国家大学信息科技大学);ETIS (UMR 8051), CY Cergy Paris University, ENSEA, CNRS(ETIS(UMR 8051),CY巴黎中央大学,ENSEA,CNRS)
Continual Learning on CLIP via Incremental Prompt Tuning with Intrinsic Textual Anchors
School of Computer Science and Engineering, University of New South Wales(新南威尔士大学计算机科学与工程学院);Data61, CSIRO(CSIRO数据61研究中心);School of Computer Science, University of Auckland(奥克兰大学计算机科学学院);Australian Institute for Machine Learning (AIML), The University of Adelaide(阿德莱德大学澳大利亚机器学习研究所)
Pengcheng laboratory(鹏城实验室);Harbin Institute of Technology(哈尔滨工业大学);Xi’an Jiaotong University(西安交通大学);Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院)
Consistent Prompting for Rehearsal-Free Continual Learning
School of Artificial Intelligence, Sun Yat-sen University(中山大学人工智能学院);Cheng Kar-Shun Robotics Institute, The Hong Kong University of Science and Technology(香港科技大学郑家纯机器人研究院);Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education(教育部机器智能与先进计算重点实验室)
Adapter-Enhanced Semantic Prompting for Continual Learning
Beijing Key Laboratory of Multimedia and Intelligent Software Technology(北京多媒体与智能软件技术重点实验室);Faculty of Information Technology, Beijing University of Technology(信息科技学院,北京理工大学);University of California at Merced(加州大学默塞德分校);Macquarie University(麦考瑞大学)
Dynamic Cross-Modal Prompt Generation for Multimodal Continual Instruction Tuning
School of Artificial Intelligence, Nanjing University(南京大学人工智能学院);State Key Laboratory for Novel Software Technology, Nanjing University(南京大学新型软件技术国家重点实验室)
Pontificia Universidad Católica de Chile(智利天主教大学);King Abdullah University of Science and Technology (KAUST)(阿卜杜拉国王科技大学);University of Pisa(比萨大学);Adobe Research(Adobe研究院)
Visual Prompt Tuning in Null Space for Continual Learning
School of Computer Science, Northwestern Polytechnical University(西北工业大学计算机学院);School of Telecommunications Engineering, Xidian University(西安电子科技大学通信工程学院)
针对持续学习中的灾难性遗忘,本文提出在 ViT 提示梯度的零空间方向进行正交投影更新,并给出理论条件与近似实现,在多个类别增量基准上优于现有方法。
C^2Prompt: Class-aware Client Knowledge Interaction for Federated Continual Learning
Wangxuan Institute of Computer Technology Peking University(北京大学王轩计算机技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Inner Mongolia University of Technology(内蒙古科技大学)
PROL : Rehearsal Free Continual Learning in Streaming Data via Prompt Online Learning
STEM University of South Australia(南澳大利亚STEM大学);Institute for Infocomm Research, A*STAR & IPAL(信息与通信研究所,A*STAR及IPAL);Systems Research Institute, Polish Academy of Sciences(波兰科学院系统研究所)
FPPL: An Efficient and Non-IID Robust Federated Continual Learning Framework
School of Computer Science and Technology, East China Normal University(华东师范大学计算机科学与技术学院);School of Software Engineering, TongJi University(同济大学软件工程学院)
Continual Learning for Segment Anything Model Adaptation
Southern University of Science and Technology(南方科技大学);City University of Hong Kong(香港城市大学);Hong Kong University of Science and Technology(香港科技大学);Tencent YouTu Lab(腾讯优图实验室)
针对动态流式数据场景,提出持续 SAM 适应基准 CoSAM 和域适配器混合算法 MoDA,通过全局特征与辅助令牌提取分离特征,实现持续分割并保持知识,优于现有方法。
Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Australian National University(澳大利亚国立大学);Monash University(莫纳什大学);Linköping University(林雪平大学)