增强视觉-语言模型的持续学习与模型融合
Enhanced Continual Learning of Vision-Language Models with Model Fusion
AI总结：
本文提出ConDU方法，通过模型融合解决VLMs在持续学习中的灾难性遗忘问题，提升零样本性能和多任务表现。
摘要：
视觉-语言模型（VLMs）通过整合视觉和文本模态，在人工智能领域取得了重大突破，实现了令人印象深刻的零样本能力。然而，当依次在多个下游任务上微调时，VLMs容易出现灾难性遗忘。现有的VLMs持续学习方法面临各种限制，通常依赖于额外的参考数据集，损害零样本性能，或局限于参数高效微调场景。在本文中，我们提出了一种新颖的持续解耦-统一（ConDU）方法，首次在VLMs中引入模型融合用于持续学习。具体而言，ConDU维护一个统一的模型，同时包含任务触发器和原型集，通过迭代过程解耦先前任务的任务专家，并将它们与新学习任务的任务专家统一。此外，我们引入了一种用于零样本场景的推理策略，通过聚合多个解耦任务专家的预测。在MTIL基准上的广泛实验表明，ConDU在所有已见任务上的平均性能相比最先进的基线提高了多达2%，同时相对于原始VLM增强了零样本能力。我们的代码可在https://github.com/zhangzicong518/ConDU上获得。
英文摘要：
Vision-Language Models (VLMs) represent a significant breakthrough in artificial intelligence by integrating visual and textual modalities to achieve impressive zero-shot capabilities. However, VLMs are susceptible to catastrophic forgetting when sequentially fine-tuned on multiple downstream tasks. Existing continual learning methods for VLMs face various limitations, often relying on additional reference datasets, compromising zero-shot performance, or being restricted to parameter-efficient fine-tuning scenarios. In this paper, we propose a novel Continual Decoupling-Unifying (ConDU) approach that pioneers the use of model fusion for continual learning in VLMs. Specifically, ConDU maintains a unified model along with task triggers and prototype sets, employing an iterative process of decoupling task experts for previous tasks and unifying them with the task expert for the newly learned task. Additionally, we introduce an inference strategy for zero-shot scenarios by aggregating predictions from multiple decoupled task experts. Extensive experiments on the MTIL benchmark show that ConDU achieves up to a 2\% improvement in average performance across all seen tasks compared to state-of-the-art baselines, while also enhancing zero-shot capabilities relative to the original VLM. Our code is available at https://github.com/zhangzicong518/ConDU.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
