留意干扰：在视觉-语言模型的参数高效持续学习中保留预训练知识
Mind the Interference: Retaining Pre-trained Knowledge in Parameter Efficient Continual Learning of Vision-Language Models
AI总结：
针对域-类别增量学习中 VLM 适应新任务时预训练知识受干扰的问题，提出 DIKI 框架，以全残差注入和分布感知校准高效保留零样本能力。
摘要：
本研究解决域-类别增量学习问题，这是一种现实但具有挑战性的持续学习场景，其中域分布和目标类别在不同任务之间都会发生变化。为处理这些多样任务，预训练视觉-语言模型（VLM）因其强大的泛化能力而被引入。然而，这带来了一个新问题：在适应新任务时，预训练 VLM 中编码的知识可能受到干扰，从而损害其固有的零样本能力。现有方法通过在额外数据集上结合知识蒸馏来微调 VLM 以解决该问题，但这需要高昂的计算开销。为高效解决此问题，我们提出了分布感知的无干扰知识整合（Distribution-aware Interference-free Knowledge Integration，DIKI）框架，从避免信息干扰的角度保留 VLM 的预训练知识。具体而言，我们设计了一种全残差机制，将新学习的知识注入冻结的骨干网络，同时对预训练知识引入最小的不利影响。此外，这种残差特性支持了我们的分布感知整合校准方案，可针对来自未见分布的测试数据显式控制信息植入过程。实验表明，我们的 DIKI 仅使用当前最先进方法 0.86% 的可训练参数，并需要显著更少的训练时间，便超过了该方法。代码可在 https://github.com/lloongx/DIKI 获取。
英文摘要：
This study addresses the Domain-Class Incremental Learning problem, a realistic but challenging continual learning scenario where both the domain distribution and target classes vary across tasks. To handle these diverse tasks, pre-trained Vision-Language Models (VLMs) are introduced for their strong generalizability. However, this incurs a new problem: the knowledge encoded in the pre-trained VLMs may be disturbed when adapting to new tasks, compromising their inherent zero-shot ability. Existing methods tackle it by tuning VLMs with knowledge distillation on extra datasets, which demands heavy computation overhead. To address this problem efficiently, we propose the Distribution-aware Interference-free Knowledge Integration (DIKI) framework, retaining pre-trained knowledge of VLMs from a perspective of avoiding information interference. Specifically, we design a fully residual mechanism to infuse newly learned knowledge into a frozen backbone, while introducing minimal adverse impacts on pre-trained knowledge. Besides, this residual property enables our distribution-aware integration calibration scheme, explicitly controlling the information implantation process for test data from unseen distributions. Experiments demonstrate that our DIKI surpasses the current state-of-the-art approach using only 0.86% of the trained parameters and requiring substantially less training time. Code is available at: https://github.com/lloongx/DIKI .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
