CLIP 与生成式潜在重放：增量学习的强基线
CLIP with Generative Latent Replay: a Strong Baseline for Incremental Learning
AI总结：
本文提出面向增量提示学习的持续生成训练方法，利用 VAE 在 CLIP 视觉嵌入空间进行类别条件生成式重放，以缓解遗忘并增强新域适配和零样本能力。
摘要：
随着 Transformers 以及 CLIP 等视觉-语言模型（VLM）的出现，微调大型预训练模型近来已成为持续学习中的一种主流策略。这促使大量提示策略被提出，用于在不引发灾难性遗忘的情况下适配基于 Transformer 的模型。然而，这些策略往往会损害预训练 CLIP 模型原有的零样本能力，并且难以适配与预训练数据显著偏离的领域。在本文中，我们提出了 Continual Generative training for Incremental prompt-Learning，这是一种简单而新颖的方法，用于在适配 CLIP 的同时缓解遗忘。简而言之，我们采用变分自编码器（VAE）在视觉编码器的嵌入空间中学习以类别为条件的分布。随后，在后续任务中，我们利用这些分布采样新的合成视觉嵌入，并训练相应的类别特定文本提示。通过在不同领域上的大量实验，我们表明，这种生成式重放方法能够适配新任务，同时提升零样本能力；评估使用了一种为 CL 场景定制的新型指标。值得注意的是，进一步分析表明，我们的方法能够缩小与联合提示微调之间的差距。代码库可在 https://github.com/aimagelab/mammoth 获取。
英文摘要：
With the emergence of Transformers and Vision-Language Models (VLMs) such as CLIP, fine-tuning large pre-trained models has recently become a prevalent strategy in Continual Learning. This has led to the development of numerous prompting strategies to adapt transformer-based models without incurring catastrophic forgetting. However, these strategies often compromise the original zero-shot capabilities of the pre-trained CLIP model and struggle to adapt to domains that significantly deviate from the pre-training data. In this work, we propose Continual Generative training for Incremental prompt-Learning, a simple and novel approach to mitigate forgetting while adapting CLIP. Briefly, we employ Variational Autoencoders (VAEs) to learn class-conditioned distributions within the embedding space of the visual encoder. We then exploit these distributions to sample new synthetic visual embeddings and train the corresponding class-specific textual prompts during subsequent tasks. Through extensive experiments on different domains, we show that such a generative replay approach can adapt to new tasks while improving zero-shot capabilities, evaluated using a novel metric tailored for CL scenarios. Notably, further analysis reveals that our approach can bridge the gap with joint prompt tuning. The codebase is available at https://github.com/aimagelab/mammoth.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
