能量结构低秩自适应持续学习
Energy-Structured Low-Rank Adaptation for Continual Learning
AI总结：
提出E²-LoRA方法，通过能量集中和排序的低秩自适应以及动态秩分配策略，解决持续学习中的任务干扰和知识压缩问题，实现最优性能。
摘要：
虽然正交子空间方法试图缓解持续学习中的任务干扰，但它们常常遭受跨基的能量扩散，阻碍知识压缩并耗尽未来任务的容量。我们观察到参数更新引起的输出特征漂移本质上是低秩的，并理论上证明沿该漂移的主方向保留参数可最小化输出重建误差。受此启发，我们提出能量集中和能量排序的低秩自适应（E²-LoRA）。通过显式地将知识排序并集中到主导秩中，E²-LoRA释放了后续任务的容量。此外，我们设计了一种动态秩分配策略，通过联合优化能量保留和模型可塑性来平衡稳定性和可塑性。在多个基准上的大量实验表明，E²-LoRA达到了最先进的性能。
英文摘要：
While orthogonal subspace methods try to mitigate task interference in Continual Learning (CL), they often suffer from energy diffusion across the basis, hindering knowledge compaction and exhausting capacity for future tasks. We observe that output feature drift induced by parameter updates is inherently low-rank, and theoretically prove that preserving parameters along the principal directions of this drift minimizes the output reconstruction error. Motivated by this, we propose \textbf{E}nergy-Concentrated and \textbf{E}nergy-Ordered \textbf{Lo}w-\textbf{R}ank \textbf{A}daptation (E$^2$-LoRA). By explicitly ordering and concentrating knowledge into leading ranks, E$^2$-LoRA frees capacity for subsequent tasks. Furthermore, we design a dynamic rank allocation strategy to balance stability and plasticity by jointly optimizing energy retention and model plasticity. Extensive experiments across multiple benchmarks demonstrate that E$^2$-LoRA achieves state-of-the-art performance. Code is available at https://github.com/kiddo127/E2-LoRA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
