防止视觉-语言模型持续学习中零样本迁移能力退化
Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Models
AI总结：
针对CLIP持续学习中零样本能力退化问题，提出ZSCL方法，通过在特征空间引入参考数据集蒸馏和在参数空间进行权重平均来防止遗忘，并在多领域任务增量学习基准上取得领先性能。
摘要：
持续学习（CL）可以帮助预训练的视觉-语言模型高效适应新的或训练不足的数据分布，而无需重新训练。然而，在对比语言-图像预训练（CLIP）模型的持续训练过程中，我们观察到由于灾难性遗忘，模型的零样本迁移能力显著退化。现有的持续学习方法可以通过回放先前数据来缓解遗忘。但是，由于CLIP数据集是私有的，回放方法无法访问预训练数据集。此外，回放先前已学习的下游任务数据虽然可以提升这些任务的性能，但代价是牺牲零样本性能。为了解决这一挑战，我们提出了一种新方法ZSCL，在特征空间和参数空间中同时防止视觉-语言模型持续学习中的零样本迁移退化。在特征空间中，我们引入一个参考数据集，用于在当前模型和初始模型之间进行蒸馏。该参考数据集应具有语义多样性，但不需要带标签、不需要在预训练中出现过，也不需要匹配的图像-文本对。在参数空间中，我们通过在训练过程中对权重进行平均来防止参数发生大幅偏移。我们提出了一个更具挑战性的多领域任务增量学习（MTIL）基准来评估不同方法，其中任务来自不同领域，而不是在单一数据集中按类别划分。我们的方法在传统的类增量学习设置和MTIL中均优于其他方法，平均得分高出9.7%。我们的代码位于https://github.com/Thunderbeee/ZSCL。
英文摘要：
Continual learning (CL) can help pre-trained vision-language models efficiently adapt to new or under-trained data distributions without re-training. Nevertheless, during the continual training of the Contrastive Language-Image Pre-training (CLIP) model, we observe that the model's zero-shot transfer ability significantly degrades due to catastrophic forgetting. Existing CL methods can mitigate forgetting by replaying previous data. However, since the CLIP dataset is private, replay methods cannot access the pre-training dataset. In addition, replaying data of previously learned downstream tasks can enhance their performance but comes at the cost of sacrificing zero-shot performance. To address this challenge, we propose a novel method ZSCL to prevent zero-shot transfer degradation in the continual learning of vision-language models in both feature and parameter space. In the feature space, a reference dataset is introduced for distillation between the current and initial models. The reference dataset should have semantic diversity but no need to be labeled, seen in pre-training, or matched image-text pairs. In parameter space, we prevent a large parameter shift by averaging weights during the training. We propose a more challenging Multi-domain Task Incremental Learning (MTIL) benchmark to evaluate different methods, where tasks are from various domains instead of class-separated in a single dataset. Our method outperforms other methods in the traditional class-incremental learning setting and the MTIL by 9.7% average score. Our code locates at https://github.com/Thunderbeee/ZSCL.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
