The "pre-training $\rightarrow$ downstream adaptation" presents both new opportunities and challenges for Continual Learning (CL). Although the recent state-of-the-art in CL is achieved through Parameter-Efficient-Tuning (PET) adaptation paradigm, only prompt has been explored, limiting its application to Transformers only. In this paper, we position prompting as one instantiation of PET, and propose a unified CL framework with general PET, dubbed as Learning-Accumulation-Ensemble (LAE). PET, e.g., using Adapter, LoRA, or Prefix, can adapt a pre-trained model to downstream tasks with fewer parameters and resources. Given a PET method, our LAE framework incorporates it for CL with three novel designs. 1) Learning: the pre-trained model adapts to the new task by tuning an online PET module, along with our adaptation speed calibration to align different PET modules, 2) Accumulation: the task-specific knowledge learned by the online PET module is accumulated into an offline PET module through momentum update, 3) Ensemble: During inference, we respectively construct two experts with online/offline PET modules (which are favored by the novel/historical tasks) for prediction ensemble. We show that LAE is compatible with a battery of PET methods and gains strong CL capability. For example, LAE with Adaptor PET surpasses the prior state-of-the-art by 1.3% and 3.6% in last-incremental accuracy on CIFAR100 and ImageNet-R datasets, respectively. Code is available at \url{https://github.com/gqk/LAE}.
Multimodal Continual Instruction Tuning with Dynamic Gradient Guidance
Harbin Institute of Technology(哈尔滨工业大学);Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)(广东人工智能与数字经济发展实验室);Chongqing Research Institute of HIT(重庆哈工大研究院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学)
School of Artificial Intelligence, UCAS(人工智能学院,UCAS);State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA(多模态人工智能系统国家重点实验室,CASIA);Centre for Artificial Intelligence and Robotics, HKISI-CAS(人工智能与机器人中心,HKISI-CAS);Huawei Inc(华为公司)
Continual Instruction Tuning for Large Multimodal Models
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);The Peng Cheng Laboratory(鹏城实验室);Wuhan AI Research(武汉人工智能研究院)
CoIN: A Benchmark of Continual Instruction tuNing for Multimodel Large Language Model
Shenzhen Institute for Advanced Study, University of Electronic Science and Technology of China(电子科技大学深圳高等研究院);University of Electronic Science and Technology of China(电子科技大学);Tongji University(同济大学)
Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning
Department of Computer Science and Technology, BNRist, Tsinghua University, Beijing, China(计算机科学与技术系,BNRist,清华大学,北京,中国);Alibaba Group, Hangzhou, China(阿里巴巴集团,杭州,中国)
Continual Learning on CLIP via Incremental Prompt Tuning with Intrinsic Textual Anchors
School of Computer Science and Engineering, University of New South Wales(新南威尔士大学计算机科学与工程学院);Data61, CSIRO(CSIRO数据61研究中心);School of Computer Science, University of Auckland(奥克兰大学计算机科学学院);Australian Institute for Machine Learning (AIML), The University of Adelaide(阿德莱德大学澳大利亚机器学习研究所)
Branch-Tuning: Balancing Stability and Plasticity for Continual Self-Supervised Learning
University of Chinese Academy of Sciences(中国科学院大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science and Innovation, Chinese Academy of Sciences(中国科学院香港创新研究院人工智能与机器人中心)
MCITlib: Multimodal Continual Instruction Tuning Library and Benchmark
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences(中国科学院大学先进交叉学科学院);State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所多模态人工智能系统国家重点实验室);Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science and Innovation, Chinese Academy of Sciences(中国科学院香港创新科学研究院人工智能与机器人中心);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Fujian Key Laboratory of Pattern Recognition and Image Understanding, School of Computer and Information Engineering, Xiamen University of Technology(福建 pattern recognition and image understanding 工程学院,厦门大学科技学院)
Don't Stop Learning: Towards Continual Learning for the CLIP Model
Xidian University(西安电子科技大学);University of Adelaide(阿德莱德大学);Australian Institute for Machine Learning(澳大利亚机器学习研究所);Shenzhen University(深圳大学);Northwestern Polytechnical University(西北工业大学)
Continual Adapter Tuning with Semantic Shift Compensation for Class-Incremental Learning
HUST AI & Visual Learning Lab (HAIV Lab)(华中科技大学人工智能与视觉学习实验室);Huazhong University of Science and Technology (HUST)(华中科技大学);Department of Computer Science(计算机科学系);Boston University(波士顿大学);ByteDance, Inc.(字节跳动公司)
Overcoming Catastrophic Forgetting in Visual Continual Learning with Reinforcement Fine-Tuning
The University of Hong Kong(香港大学);The Hong Kong University of Science and Technology(香港科学与技术大学);Hong Kong Generative AI Research and Development Center(香港生成式人工智能研究与开发中心)
Dynamic Cross-Modal Prompt Generation for Multimodal Continual Instruction Tuning
School of Artificial Intelligence, Nanjing University(南京大学人工智能学院);State Key Laboratory for Novel Software Technology, Nanjing University(南京大学新型软件技术国家重点实验室)
Visual Prompt Tuning in Null Space for Continual Learning
School of Computer Science, Northwestern Polytechnical University(西北工业大学计算机学院);School of Telecommunications Engineering, Xidian University(西安电子科技大学通信工程学院)
针对持续学习中的灾难性遗忘,本文提出在 ViT 提示梯度的零空间方向进行正交投影更新,并给出理论条件与近似实现,在多个类别增量基准上优于现有方法。
SLCA++: Unleash the Power of Sequential Fine-tuning for Continual Learning with Pre-training
Australian Artificial Intelligence Institute(澳大利亚人工智能研究所);University of Technology Sydney(悉尼科技大学);Tsinghua University(清华大学);Beihang University(北京航空航天大学);Institute of Information Science, Beijing Jiaotong University(北京交通大学信息科学研究所)
Continual Learning with Elastic Regularization and Synthetic Replay for Federated MLLM Fine-Tuning
The University of British Columbia(英属哥伦比亚大学);Fudan University(复旦大学);Royal College of Science, Imperial College London(伦敦帝国理工学院皇家科学学院);Dyson School of Design Engineering(戴森设计工程学院);Suzhou Institute of Biomedical Engineering and Technology (SIBET), Chinese Academy of Sciences(中国科学院苏州生物医学工程技术研究所);East China Normal University(华东师范大学)
ContiStain: Cross-Domain Relation-Preserving Distillation for Continual Multi-Domain Virtual IHC Staining
School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳)计算机科学与技术学院);School of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系)
Multi-Scale Global-Instance Prompt Tuning for Continual Test-time Adaptation in Medical Image Segmentation
School of Computer Science, University of Nottingham, UK(诺丁汉大学计算机科学学院);School of Artificial Intelligence, Shenzhen University, China(深圳大学人工智能学院);Department of Information Engineering and Computer Science, University of Trento, Italy(特伦托大学信息工程与计算机科学系);School of Computer Science and Information Engineering, Hefei University of Technology, China(合肥工业大学计算机科学与信息工程学院)
ModalPrompt: Towards Efficient Multimodal Continual Instruction Tuning with Dual-Modality Guided Prompt
State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA(多模态人工智能系统国家重点实验室);School of Artificial Intelligence, UCAS(人工智能学院);Centre for Artificial Intelligence and Robotics, HKISI-CAS(人工智能与机器人中心)
MM-Prompt: Cross-Modal Prompt Tuning for Continual Visual Question Answering
Khoury College of Computer Sciences, Northeastern University(东北大学克劳尔计算机科学学院);New Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所模式识别新实验室)
Hyper-LLaVA: Hyperbolic Uncertainty-aware Modality-Balanced Routing for Multimodal Continual Instruction Tuning
Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机技术研究所);Institute of Software Chinese Academy of Sciences(中国科学院软件研究所);University of Chinese Academy of Sciences(中国科学院大学)
Continual Adaptation: Environment-Conditional Parameter Generation for Object Detection in Dynamic Scenarios
College of Intelligence and Computing, Tianjin University(智能与计算学院,天津大学);School of Electronic Engineering, Xidian University(电子工程学院,西安电子科技大学);Peng Cheng Laboratory(鹏城实验室)
Progressive Multimodal Alignment for Continual Instruction Tuning
Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences(中国科学院脑科学与智能技术卓越创新中心);Kyoto University(京都大学);Migu Culture Technology Co.,Ltd.(咪咕文化科技有限公司);State Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology(脑认知与类脑智能技术国家重点实验室)