分解与组合：通过单个LoRA中的秩-1专家池实现高效的视觉-语言连续学习
Decomposing and Composing: Towards Efficient Vision-Language Continual Learning via Rank-1 Expert Pool in a Single LoRA
AI总结：
通过单个LoRA中的秩-1专家池实现高效的视觉-语言连续学习，减少参数更新并提升领域感知能力。
摘要：
在视觉-语言模型（VLMs）中，连续学习（CL）面临显著挑战，即在提高任务适应性和避免灾难性遗忘方面存在困难。现有方法通常具有较大的推理负担或依赖外部知识，而低秩适应（LoRA）通过实现参数高效调优显示出潜力。然而，直接使用LoRA来缓解灾难性遗忘问题并不简单，我们引入了一个新的框架，将单个LoRA模块重新结构化为可分解的秩-1专家池。我们的方法通过从该专家池中选择，动态组合稀疏、任务特定的更新，由[CLS]标记的语义引导。此外，我们提出了一种激活引导正交（AGO）损失，通过正交化LoRA权重的关键部分跨任务。这种稀疏组合和正交化使得参数更新更少，从而实现领域感知学习，同时最小化跨任务干扰并保持下游任务性能。在多个设置下的广泛实验显示，该方法在所有指标上均取得最先进的结果，超越了零样本上限的泛化能力。值得注意的是，与基线方法相比，它将可训练参数减少了96.7%，消除了对外部数据集或任务-ID判别器的依赖。合并的LoRAs保留了较少的权重且无推理延迟，使我们的方法计算轻量。
英文摘要：
Continual learning (CL) in vision-language models (VLMs) faces significant challenges in improving task adaptation and avoiding catastrophic forgetting. Existing methods usually have heavy inference burden or rely on external knowledge, while Low-Rank Adaptation (LoRA) has shown potential in reducing these issues by enabling parameter-efficient tuning. However, considering directly using LoRA to alleviate the catastrophic forgetting problem is non-trivial, we introduce a novel framework that restructures a single LoRA module as a decomposable Rank-1 Expert Pool. Our method learns to dynamically compose a sparse, task-specific update by selecting from this expert pool, guided by the semantics of the [CLS] token. In addition, we propose an Activation-Guided Orthogonal (AGO) loss that orthogonalizes critical parts of LoRA weights across tasks. This sparse composition and orthogonalization enable fewer parameter updates, resulting in domain-aware learning while minimizing inter-task interference and maintaining downstream task performance. Extensive experiments across multiple settings demonstrate state-of-the-art results in all metrics, surpassing zero-shot upper bounds in generalization. Notably, it reduces trainable parameters by 96.7% compared to the baseline method, eliminating reliance on external datasets or task-ID discriminators. The merged LoRAs retain less weights and incur no inference latency, making our method computationally lightweight.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
