持续学习的语义残差提示
Semantic Residual Prompts for Continual Learning
AI总结：
针对持续学习提示选择易受灾难性遗忘的问题，提出利用CLIP的两级提示选择机制和残差语义迁移，显著优于现有方法及零样本CLIP。
摘要：
针对持续学习（CL）的提示调优方法冻结大型预训练模型，并训练少量称为提示的参数向量。大多数此类方法将这些向量组织在键值对池中，并使用输入图像作为查询来检索提示（值）。然而，由于键在任务进展过程中被学习，提示选择策略本身也容易遭受灾难性遗忘，这一问题常被现有方法忽视。例如，为适应新任务而引入的提示可能最终干扰先前学习的提示。为使选择策略更加稳定，我们利用基础模型（CLIP）在两级适应机制中选择提示。具体而言，第一级利用标准文本提示池用于CLIP文本编码器，从而产生稳定的类原型。第二级则使用这些原型连同查询图像作为键来索引第二个池。检索到的提示用于适应预训练的ViT，赋予可塑性。在此过程中，我们还提出了一种新颖的残差机制，将CLIP语义迁移到ViT层。通过在已建立的CL基准上的广泛分析，我们表明我们的方法显著优于最先进的CL方法和零样本CLIP测试。值得注意的是，即使对于与骨干模型预训练知识存在显著域差距的数据集，我们的发现也成立，这一点通过在卫星图像和医学数据集上的实验得以展示。代码库可在https://github.com/aimagelab/mammoth获取。
英文摘要：
Prompt-tuning methods for Continual Learning (CL) freeze a large pre-trained model and train a few parameter vectors termed prompts. Most of these methods organize these vectors in a pool of key-value pairs and use the input image as query to retrieve the prompts (values). However, as keys are learned while tasks progress, the prompting selection strategy is itself subject to catastrophic forgetting, an issue often overlooked by existing approaches. For instance, prompts introduced to accommodate new tasks might end up interfering with previously learned prompts. To make the selection strategy more stable, we leverage a foundation model (CLIP) to select our prompts within a two-level adaptation mechanism. Specifically, the first level leverages a standard textual prompt pool for the CLIP textual encoder, leading to stable class prototypes. The second level, instead, uses these prototypes along with the query image as keys to index a second pool. The retrieved prompts serve to adapt a pre-trained ViT, granting plasticity. In doing so, we also propose a novel residual mechanism to transfer CLIP semantics to the ViT layers. Through extensive analysis on established CL benchmarks, we show that our method significantly outperforms both state-of-the-art CL approaches and the zero-shot CLIP test. Notably, our findings hold true even for datasets with a substantial domain gap w.r.t. the pre-training knowledge of the backbone model, as showcased by experiments on satellite imagery and medical datasets. The codebase is available at https://github.com/aimagelab/mammoth.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
