arXiv 2603.112112026-08-13cs.CVcs.AI

通过非线性多适配器的视觉-语言模型实现简单的效率增量学习框架

A Simple Efficiency Incremental Learning Framework via Vision-Language Model with Nonlinear Multi-Adapters

Faculty of Information Technology, University of Jyväskylä（信息科技学院，于韦斯屈耶大学）；National University of Singapore（新加坡国立大学）；Lab of Brain-Machine Intelligence, Zhejiang University（脑机智能实验室，浙江大学）；School of Computer Science and Technology, Dalian University of Technology（计算机科学与技术学院，大连理工大学）

Haihua Luo, Xuming Ran, Jiangrong Shen, Timo Hämäläinen, Zhonghua Chen, Qi Xu, Fengyu Cong

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出SimE框架，利用视觉-语言模型与适配器提升增量学习效率，发现适配器连接数与模型能力呈非线性关系，实验表明在TinyImageNet和CIFAR-100上优于传统方法。

摘要：

增量学习（IL）旨在在保留先前知识的同时学习新任务。将预训练视觉-语言模型的零样本学习能力整合到IL方法中标志着重大进展。然而，这些方法面临三个主要挑战：（1）需要提高训练效率；（2）依赖内存银行存储先前数据；（3）需要强大的主干网络来增强模型能力。本文提出SimE，一种简单高效的框架，采用专为IL任务设计的视觉-语言模型与适配器。我们发现适配器连接数与模型IL能力之间存在非线性相关性。虽然在Transformer块之间增加适配器连接可提升模型性能，但在较小的增量步骤中在Transformer块内增加适配器连接可能不会增强甚至会降低模型的IL能力。大量实验结果表明，SimE在TinyImageNet上优于传统方法9.6%，在CIFAR-100上优于其他CLIP方法5.3%。此外，我们进行系统研究以提高CLIP的零样本能力的利用。我们建议将SimE的编码器替换为在更大数据集（如LAION2B）和更强架构（如ViT-L/14）上训练的CLIP模型。

英文摘要：

Incremental Learning (IL) aims to learn new tasks while preserving previously acquired knowledge. Integrating the zero-shot learning capabilities of pre-trained vision-language models into IL methods has marked a significant advancement. However, these methods face three primary challenges: (1) the need for improved training efficiency; (2) reliance on a memory bank to store previous data; and (3) the necessity of a strong backbone to augment the model's capabilities. In this paper, we propose SimE, a Simple and Efficient framework that employs a vision-language model with adapters designed specifically for the IL task. We report a remarkable phenomenon: there is a nonlinear correlation between the number of adaptive adapter connections and the model's IL capabilities. While increasing adapter connections between transformer blocks improves model performance, adding more adaptive connections within transformer blocks during smaller incremental steps does not enhance, and may even degrade the model's IL ability. Extensive experimental results show that SimE surpasses traditional methods by 9.6% on TinyImageNet and outperforms other CLIP-based methods by 5.3% on CIFAR-100. Furthermore, we conduct a systematic study to enhance the utilization of the zero-shot capabilities of CLIP. We suggest replacing SimE's encoder with a CLIP model trained on larger datasets (e.g., LAION2B) and stronger architectures (e.g., ViT-L/14).

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。