通过内在文本锚点进行增量提示调优实现CLIP的持续学习
Continual Learning on CLIP via Incremental Prompt Tuning with Intrinsic Textual Anchors
AI总结：
本文提出基于增量提示调优的CLIP持续学习方法，通过引入文本原型作为稳定锚点，提升多模态嵌入空间稳定性，并联合优化视觉和文本提示以减少遗忘。
摘要：
持续学习（CL）使深度网络能够在避免灾难性遗忘的同时获取新知识。预训练模型（PTMs）的强大泛化能力，如对比语言-图像预训练（CLIP）模型，已激发了多种针对新和专门任务的CL方法，提供了丰富的多模态嵌入，支持轻量级的增量提示调优。现有方法往往依赖于基于特定假设的复杂设计，如复杂的提示池正则化方案、专门的路由机制或多阶段增量，这些引入了额外且可能不必要的复杂性，未能充分利用CLIP的内在能力。在本文中，我们提出了一种基于增量提示调优的简洁CL方法，充分利用其多模态结构和文本表示的稳定性。我们的方法，文本原型引导的提示调优（TPPT），将文本原型不仅作为静态分类器（如现有方法），而是作为稳定的锚点来引导视觉提示的学习，从而塑造嵌入空间（即TPPT-V）。我们证明了我们的双向监督策略在减少遗忘的同时更有效地学习新知识。为进一步缩小视觉-语言差距，我们在CL中联合优化视觉和文本提示（即TPPT-VT）。我们还引入了对文本锚点的关联多样性正则化，以防止嵌入空间坍塌并缓解相关遗忘。广泛的实验和分析证明了我们所提出方法的有效性，突显了利用CLIP内在指导进行持续适应的好处。
英文摘要：
Continual learning (CL) enables deep networks to acquire new knowledge while avoiding catastrophic forgetting. The powerful generalization ability of pre-trained models (PTMs), such as the Contrastive Language-Image Pre-training (CLIP) model, has inspired a range of CL methods targeting new and specialized tasks, providing rich multi-modal embeddings that support lightweight, incremental prompt tuning. Existing methods often rely on complex designs built upon specific assumptions, such as intricate regularization schemes for prompt pools, specialized routing mechanisms, or multi-stage incrementations, that introduce additional-and possibly unnecessary-complexity, underutilizing CLIP's intrinsic capabilities. In this paper, we propose a concise CL approach for CLIP based on incremental prompt tuning that fully exploits its multi-modal structure and the stability of textual representations. Our method, Textual Prototype-guided Prompt Tuning (TPPT), introduces textual prototypes not merely as static classifiers, as in existing methods, but as stable anchors to guide the learning of visual prompts, thereby shaping the embedding space (i.e., TPPT-V). We show that our bidirectional supervision strategy enables more effective learning of new knowledge while reducing forgetting. To further close the vision-language gap during CL, we jointly optimizes visual and textual prompts (i.e., TPPT-VT). We also introduce a relational diversity regularization on the textual anchors to prevent embedding space collapse and mitigate correlated forgetting. Extensive experiments and analyses demonstrate the effectiveness of our proposed approach, highlighting the benefits of leveraging CLIP's intrinsic guidance for continual adaptation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
