Mind the Gap：在基于CLIP的持续学习中保留并补偿模态差距
Mind the Gap: Preserving and Compensating for the Modality Gap in CLIP-Based Continual Learning
AI总结：
本文提出MG-CLIP方法，通过保留模态差距缓解遗忘并补偿模态差距增强新数据容量，提升了CLIP在类增量学习中的性能，且无需额外重放数据。
摘要：
持续学习旨在使模型能够从持续到来的数据中按顺序学习，同时保持对先前学习任务的性能。随着对比语言-图像预训练模型（CLIP）在各种下游任务中展现出强大能力，人们越来越有兴趣在此类场景中利用CLIP进行持续学习。大多数现有工作忽略了CLIP中固有的模态差距，这是其泛化能力和适应性的关键因素。在本文中，我们分析了视觉-语言预训练模型微调过程中模态差距的变化。我们的观察揭示，模态差距有效地反映了预训练知识被保留的程度。基于这些见解，我们提出了一种简单而有效的方法，MG-CLIP，该方法提高了CLIP在类增量学习中的性能。我们的方法利用模态差距保留来缓解遗忘，并利用模态差距补偿来增强对新数据的容量，为持续学习引入了一种新颖的基于模态差距的视角。在多个基准上的大量实验表明，我们的方法优于现有方法，且无需额外的重放数据。我们的代码可在https://github.com/linlany/MindtheGap获取。
英文摘要：
Continual learning aims to enable models to learn sequentially from continuously incoming data while retaining performance on previously learned tasks. With the Contrastive Language-Image Pre-trained model (CLIP) exhibiting strong capabilities across various downstream tasks, there has been growing interest in leveraging CLIP for continual learning in such scenarios. Most existing works overlook the inherent modality gap in CLIP, a key factor in its generalization and adaptability. In this paper, we analyze the variations in the modality gap during the fine-tuning of vision-language pre-trained models. Our observations reveal that the modality gap effectively reflects the extent to which pre-trained knowledge is preserved. Based on these insights, we propose a simple yet effective method, MG-CLIP, that improves CLIP's performance in class-incremental learning. Our approach leverages modality gap preservation to mitigate forgetting and modality gap compensation to enhance the capacity for new data, introducing a novel modality-gap-based perspective for continual learning. Extensive experiments on multiple benchmarks demonstrate that our method outperforms existing approaches without requiring additional replay data. Our code is available at https://github.com/linlany/MindtheGap.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
