arXiv 2501.150402025-01-25cs.CV

用于少样本分类的视觉语言模型互补子空间低秩适应

Complementary Subspace Low-Rank Adaptation of Vision-Language Models for Few-Shot Classification

School of EECE, University of Chinese Academy of Science（中国科学院大学电子电气与通信工程学院）；Dolby Lab. Inc.（杜比实验室公司）

Zhongqi Wang, Jia Dai, Kai Li, Xu Li, Yanmeng Guo, Maosheng Xiang

备注：: Preprint version

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出Comp-LoRA方法，通过在互补子空间优化低秩矩阵缓解灾难性遗忘，在少样本分类中提升了准确率并保留了VLM零样本性能。

摘要：

视觉语言模型（VLM）被设计用于大规模图文对齐，作为预训练基础模型。对于下游少样本分类任务，参数高效微调（PEFT）VLM在计算机视觉领域获得了广泛关注。已有研究探讨了提示微调和线性适配器等PEFT方法来微调VLM，但低秩适应（LoRA）算法却很少被考虑用于少样本微调VLM。将LoRA用于少样本微调的主要障碍是灾难性遗忘问题。因为视觉语言对齐知识对少样本学习的泛化性很重要，而低秩适应会干扰预训练权重矩阵中信息量最大的方向。我们提出了互补子空间低秩适应（Comp-LoRA）方法，以在少样本VLM微调中缓解灾难性遗忘问题。具体而言，我们在互补子空间中优化低秩矩阵，从而在学习新的少样本信息时保留VLM的通用视觉语言对齐能力。我们对所提出的Comp-LoRA方法和其他PEFT方法在少样本分类微调VLM上进行了对比实验。我们还展示了与直接将LoRA应用于VLM相比，所提方法对灾难性遗忘问题的抑制作用。结果表明，所提方法在Top-1准确率上比基线方法高出约+1.0%，并且在Top-1准确率上比基线方法保留了约+1.3%的VLM零样本性能。

英文摘要：

Vision language model (VLM) has been designed for large scale image-text alignment as a pretrained foundation model. For downstream few shot classification tasks, parameter efficient fine-tuning (PEFT) VLM has gained much popularity in the computer vision community. PEFT methods like prompt tuning and linear adapter have been studied for fine-tuning VLM while low rank adaptation (LoRA) algorithm has rarely been considered for few shot fine-tuning VLM. The main obstacle to use LoRA for few shot fine-tuning is the catastrophic forgetting problem. Because the visual language alignment knowledge is important for the generality in few shot learning, whereas low rank adaptation interferes with the most informative direction of the pretrained weight matrix. We propose the complementary subspace low rank adaptation (Comp-LoRA) method to regularize the catastrophic forgetting problem in few shot VLM finetuning. In detail, we optimize the low rank matrix in the complementary subspace, thus preserving the general vision language alignment ability of VLM when learning the novel few shot information. We conduct comparison experiments of the proposed Comp-LoRA method and other PEFT methods on fine-tuning VLM for few shot classification. And we also present the suppression on the catastrophic forgetting problem of our proposed method against directly applying LoRA to VLM. The results show that the proposed method surpasses the baseline method by about +1.0\% Top-1 accuracy and preserves the VLM zero-shot performance over the baseline method by about +1.3\% Top-1 accuracy.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。