arXiv 2406.188682024-12-18cs.CV

提升视觉语言模型持续学习中的跨域判别性

Advancing Cross-domain Discriminability in Continual Learning of Vision-Language Models

Institute of Science Tokyo（东京科学研究院）；University of California Berkeley（加州大学伯克利分校）；Nanyang Technological University（南洋理工大学）；South China University of Technology（华南理工大学）；Greater Bay Area Institute for Innovation, Hunan University（湖南大学粤港澳大湾区创新研究院）

Yicheng Xu, Yuxin Chen, Jiahao Nie, Yusong Wang, Huiping Zhuang, Manabu Okumura

备注：: Accepted by NeurIPS 2024

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出RAIL方法，利用递归岭回归适配器与免训练融合模块，在无需参考数据和域身份提示下实现VLM的持续学习，绝对保留零样本能力并提升跨域判别性。

摘要：

视觉语言模型（VLMs）的持续学习（CL）克服了仅关注先前遇到类的传统CL的限制。在VLMs的CL过程中，我们不仅需要防止对增量学习知识的灾难性遗忘，还需要保留VLMs的零样本能力。然而，现有方法需要额外的参考数据集来维持这种零样本能力，并依赖域身份提示来跨不同域对图像进行分类。在本研究中，我们提出了基于回归的解析增量学习（RAIL），它利用基于递归岭回归的适配器以不遗忘的方式从一系列域中学习，并通过将特征投影到更高维空间来解耦跨域相关性。配合免训练融合模块，RAIL在没有任何参考数据的情况下绝对保留了VLM在未见域上的零样本能力。此外，我们引入了跨域任务无关增量学习（X-TAIL）设置。在此设置中，要求CL学习器从多个域增量学习，并在没有任何域身份提示的情况下对来自已见和未见域的测试图像进行分类。我们在理论上证明了RAIL对增量学习域的绝对记忆。实验结果证实了RAIL在X-TAIL和现有的多域任务增量学习设置中的最先进性能。代码发布在https://github.com/linghan1997/Regression-based-Analytic-Incremental-Learning。

英文摘要：

Continual learning (CL) with Vision-Language Models (VLMs) has overcome the constraints of traditional CL, which only focuses on previously encountered classes. During the CL of VLMs, we need not only to prevent the catastrophic forgetting on incrementally learned knowledge but also to preserve the zero-shot ability of VLMs. However, existing methods require additional reference datasets to maintain such zero-shot ability and rely on domain-identity hints to classify images across different domains. In this study, we propose Regression-based Analytic Incremental Learning (RAIL), which utilizes a recursive ridge regression-based adapter to learn from a sequence of domains in a non-forgetting manner and decouple the cross-domain correlations by projecting features to a higher-dimensional space. Cooperating with a training-free fusion module, RAIL absolutely preserves the VLM's zero-shot ability on unseen domains without any reference data. Additionally, we introduce Cross-domain Task-Agnostic Incremental Learning (X-TAIL) setting. In this setting, a CL learner is required to incrementally learn from multiple domains and classify test images from both seen and unseen domains without any domain-identity hint. We theoretically prove RAIL's absolute memorization on incrementally learned domains. Experiment results affirm RAIL's state-of-the-art performance in both X-TAIL and existing Multi-domain Task-Incremental Learning settings. The code is released at https://github.com/linghan1997/Regression-based-Analytic-Incremental-Learning.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。