谱不平衡导致低秩持续适应中的遗忘
Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptation
AI总结：
本文从知识分解角度发现低秩适应存在奇异值谱不平衡，导致遗忘，并提出在受限Stiefel流形上解耦任务更新幅度与方向结构的约束优化方法，有效缓解前后向遗忘。
摘要：
参数高效的持续学习旨在使预训练模型适应顺序任务而不遗忘先前获得的知识。大多数现有方法将持续学习视为避免与过去更新干扰，而非考虑当前任务特定更新的哪些属性能够自然地保留先前获得的知识。从知识分解的角度，我们观察到低秩适应表现出高度不平衡的奇异值谱：少数主导成分吸收了大部分适应能量，从而（i）更可能破坏先前获得的知识，以及（ii）使更新更容易受到后续任务干扰。为了实现组件之间的显式平衡，我们将任务更新的幅度与其方向结构解耦，并将其表述为受限Stiefel流形上的约束优化问题。我们使用与视觉语言模型中使用的标准深度学习优化器兼容的投影一阶方法来解决这个问题。我们的方法减轻了前后向遗忘，持续优于持续学习基线。实现代码可从此https URL获取。
英文摘要：
Parameter-efficient continual learning aims to adapt pre-trained models to sequential tasks without forgetting previously acquired knowledge. Most existing approaches treat continual learning as avoiding interference with past updates, rather than considering what properties make the current task-specific update naturally preserve previously acquired knowledge. From a knowledge-decomposition perspective, we observe that low-rank adaptations exhibit highly imbalanced singular value spectra: a few dominant components absorb most of the adaptation energy, thereby (i) more likely to disrupt previously acquired knowledge and (ii) making the update more vulnerable to interference from subsequent tasks. To enable explicit balance among components, we decouple the magnitude of the task update from its directional structure and formulate it as a constrained optimization problem on a restricted Stiefel manifold. We address this problem using a projected first-order method compatible with standard deep-learning optimizers used in vision-language models. Our method mitigates both backward and forward forgetting, consistently outperforming continual learning baselines. The implementation code is available at https://github.com/haodotgu/EBLoRA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
