当一个适配器代表多个任务：在持续微调中发现低秩冗余
When One Adapter Speaks for Many: Discovering Low-Rank Redundancy in Continual Fine-Tuning
AI总结：
本文发现持续学习中的LoRA适配器存在显著低秩冗余，提出LiteLoRA门控机制，通过复用现有适配器减少20-70%活跃适配器，同时保持或超越现有性能。
摘要：
低秩适配（LoRA）已成为大型预训练模型参数高效微调的标准工具。在持续学习（CL）中按顺序应用于任务时，标准假设是每个新任务需要一个专用的低秩适配器。在这项工作中，我们从经验和结构上挑战了这一假设。我们表明，CL中的任务特定LoRA适配器表现出显著的低秩冗余：不同任务上训练的适配器所张成的子空间大量重叠，并且在许多情况下，较早的适配器可以忠实地表示较晚的任务。基于这一观察，我们提出了LiteLoRA，一种即插即用的门控机制，它在训练时学习是招募新适配器还是重用现有的低秩表示。我们的方法将活跃适配器的数量减少了20-70%，同时在标准CL基准测试中达到或超越了最先进的性能，揭示了结构冗余是普遍存在的，并且选择性学习足以在不牺牲可塑性的情况下实现稳定性。
英文摘要：
Low-Rank Adaptation (LoRA) has become the standard tool for parameter-efficient fine-tuning of large pretrained models. When applied sequentially across tasks in Continual Learning (CL), the standard assumption is that each new task requires a dedicated low-rank adapter. In this work, we challenge this assumption empirically and structurally. We show that task-specific LoRA adapters in CL exhibit significant low-rank redundancy: the subspaces spanned by adapters trained on different tasks substantially overlap, and in many cases earlier adapters can faithfully represent later tasks. Building on this observation, we propose LiteLoRA, a plug-and-play gating mechanism that learns at train time whether to recruit a new adapter or reuse existing low-rank representations. Our method reduces the number of active adapters by 20-70% while matching or exceeding state-of-the-art performance on standard CL benchmarks, revealing that structural redundancy is pervasive and that selective learning is sufficient to achieve stability without sacrificing plasticity.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
