对比语言-图像预训练（CLIP）通过在共享嵌入空间中对齐视觉和文本模态，展现出显著的零样本泛化能力。然而，当在多样化任务上持续微调时，CLIP 会遭遇灾难性遗忘，并且其嵌入对齐会退化，从而削弱其零样本能力。在本文中，我们提出梯度零空间投影（GNSP），这是一种高效的持续学习方法，将特定任务梯度投影到先前所学知识的零空间中。该正交投影在数学上防止了对先前任务的干扰，且无需依赖回放或架构修改。此外，为保持 CLIP 固有的泛化特性，我们引入知识蒸馏，并将其与受 CLIP 预训练启发的模态对齐保持损失相结合，以在微调期间稳定多模态嵌入空间的结构。在由 11 个任务组成的 MTIL 基准上，我们的方法在 Average 和 Last 两个关键指标上均取得 SOTA 性能。更重要的是，实验表明，该方法成功保持了 CLIP 原有的模态间隙和跨模态检索性能，证实了其在整个持续学习过程中维持鲁棒视觉-语言空间的有效性。
英文摘要：
Contrastive Language-Image Pretraining has demonstrated remarkable zero-shot generalization by aligning visual and textual modalities in a shared embedding space. However, when continuously fine-tuned on diverse tasks, CLIP suffers from catastrophic forgetting and degradation of its embedding alignment, undermining its zero-shot capabilities. In this work, we propose Gradient Null Space Projection (GNSP), an efficient continual learning method that projects task-specific gradients onto the null space of previously learned knowledge. This orthogonal projection mathematically prevents interference with previous tasks without relying on rehearsal or architectural modification. Furthermore, to preserve the inherent generalization property of CLIP, we introduce knowledge distillation and combine it with a modality alignment preservation loss inspired by CLIP pre-training to stabilize the structure of the multimodal embedding space during fine-tuning. On the MTIL benchmark consisting of 11 tasks, our method achieved SOTA performance on both the Average and Last key metrics. More importantly, experiments show that our method successfully maintains the original modality gap and cross-modal retrieval performance of CLIP, confirming its effectiveness in maintaining a robust visual-language space throughout the continual learning process.
Tianjin University（天津大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The University of Sheffield（谢菲尔德大学）
Vision Language Models (VLMs), pre-trained on large-scale image-text datasets, enable zero-shot predictions for unseen data but may underperform on specific unseen tasks. Continual learning (CL) can help VLMs effectively adapt to new data distributions without joint training, but faces challenges of catastrophic forgetting and generalization forgetting. Although significant progress has been achieved by distillation-based methods, they exhibit two severe limitations. One is the popularly adopted single-teacher paradigm fails to impart comprehensive knowledge, The other is the existing methods …
Beijing University of Posts and Telecommunications（北京邮电大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；West China Hospital, Sichuan University（四川大学华西医院）；Shanghai Jiao Tong University（上海交通大学）；Sichuan University Pittsburgh Institute（四川大学匹兹堡学院）
Inevitable domain and task discrepancies in real-world scenarios can impair the generalization performance of the pre-trained deep models for medical data. Therefore, we audaciously propose that we should build a general-purpose medical AI system that can be seamlessly adapted to downstream domains/tasks. Since the domain/task adaption procedures usually involve additional labeling work for the target data, designing a data-efficient adaption algorithm is desired to save the cost of transferring the learned knowledge. Our recent work found that vision-language models (VLMs) are efficient learn…
Bisecle: Binding and Separation in Continual Learning for Video Language Understanding
School of Computer Science University of New South Wales（计算机科学学院新南威尔士大学）；School of Computer Science and Engineering University of New South Wales（计算机科学与工程学院新南威尔士大学）；DEVCOM Army Research Laboratory（陆军研究实验室）
Advancing Cross-domain Discriminability in Continual Learning of Vision-Language Models
Institute of Science Tokyo（东京科学研究院）；University of California Berkeley（加州大学伯克利分校）；Nanyang Technological University（南洋理工大学）；South China University of Technology（华南理工大学）；Greater Bay Area Institute for Innovation, Hunan University（湖南大学粤港澳大湾区创新研究院）
CalFuse: Multi-Modal Continual Learning via Feature Calibration and Parameter Fusion
College of Intelligent Robotics and Advanced Manufacturing, Fudan University（智能机器人与先进制造学院，复旦大学）；School of Future Science and Engineering, Soochow University（未来科学与工程学院，苏州大学）；DataLab: Data Science and Informatics, University of California, Davis（数据实验室：数据科学与信息学，加州大学戴维斯分校）；University of Rochester（罗切斯特大学）；Ningbo University（宁波大学）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Beijing University of Posts and Telecommunications（北京邮电大学）；Academy for Computer Science and Informatics, Cardiff University（计算机科学与信息学学院，卡迪夫大学）
Instruction-Grounded Visual Projectors for Continual Learning of Generative Vision-Language Models
School of Computer Science and Engineering, Chung-Ang University（Chung-Ang 大学计算机科学与工程学院）；School of Computer Science, University of Birmingham（布拉德福德大学计算机科学学院）
Forging a Dynamic Memory: Retrieval-Guided Continual Learning for Generalist Medical Foundation Models
College of Intelligent Robotics and Advanced Manufacturing（智能机器人与先进制造学院）；Fudan University（复旦大学）；Fysics Intelligence Technologies Co., Ltd. (Fysics AI)（Fysics智能技术有限公司（Fysics AI））；School of Computer Science and Engineering（计算机科学与工程学院）；Central South University（中南大学）
Synthetic Data is an Elegant GIFT for Continual Vision-Language Models
School of Computer Science, Wuhan University（武汉大学计算机学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Wuhan AI Research（武汉人工智能研究院）；Taikang Center for Life and Medical Sciences, Wuhan University（武汉大学泰康生命医学中心）
VLM-C4L: Continual Core Dataset Learning with Corner Case Optimization via Vision-Language Models for Autonomous Driving
City University of Hong Kong（香港城市大学）；Soochow University（苏州大学）；McGill University（麦吉尔大学）；Hon Hai Research Institute（鸿海研究院）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
DOTA: Distributional Test-Time Adaptation of Vision-Language Models
State Key Laboratory of Networking and Switching Technology（网络与交换技术国家重点实验室）；Beijing University of Posts and Telecommunications（北京邮电大学）；College of Intelligence and Computing（智能与计算学院）；Tianjin University（天津大学）；School of Computer Science and Technology（计算机科学与技术学院）；Harbin Institute of Technology Shenzhen（哈尔滨工业大学深圳学院）；Institute for Infocomm Research, A*STAR（信息通信研究机构，A*STAR）；Show Lab, National University of Singapore（新加坡国立大学Show实验室）；National University of Singapore（新加坡国立大学）
