arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.16664cs.LG
收藏

使LoRA可识别:面向持续学习的正交对齐

Making LoRA Identifiable: Orthogonal Alignment for Continual Learning

Sony Research India(索尼印度研究院)

Pratik Rakesh Singh, Mohammadi Zaki, Akash Saha, Aneesh Mukkamala, Pankaj Wasnik

AI总结:

针对现有持续LoRA学习方法内存开销大、表示对齐不足的问题,提出SOAR框架,通过Stiefel流形约束A矩阵与正交Procrustes对齐B矩阵,在无额外内存开销下实现优异的持续学习性能。

摘要:

将参数高效微调(PEFT)应用于持续学习的兴趣日益浓厚,因为它能实现更快的训练、缓解灾难性遗忘并促进对新任务的更好适应。现有方法探索了一系列策略,包括为每个传入任务扩展新的LoRA分支、在所有特定任务的LoRA模块下基于正交损失目标进行训练,以及采用门控机制整合新旧LoRA模块。然而,这些方法的内存成本随任务数量线性增长,限制了其可扩展性。更近的方法研究利用LoRA参数化中的不对称性来训练单个LoRA,但它们往往存在LoRA参数空间中表示对齐不足的问题。为解决这些局限,我们提出Stiefel优化与对齐旋转(SOAR),这是一个用于持续LoRA学习的两阶段框架。首先,我们将A矩阵约束在Stiefel流形上,从而减少与正交旋转矩阵空间的坐标不对齐。其次,我们为B矩阵引入正交Procrustes对齐过程以估计最优旋转,同时结合按秩保留合并策略,该策略可量化新任务与先前学习任务表示之间的方向一致性。我们在多个模型和基准上开展了广泛实验,证明SOAR在不产生额外内存开销的情况下实现了出色的持续学习性能。

英文摘要:

There is growing interest in applying Parameter-Efficient Fine-Tuning (PEFT) to Continual Learning, as it enables faster training, mitigates catastrophic forgetting, and facilitates better adaptation to new tasks. Prior approaches have explored a range of strategies, including extending new LoRA branches for each incoming task, training under orthogonal loss objectives across all task-specific LoRA modules, and employing gating mechanisms to integrate new and existing LoRA modules. However, these methods incur memory costs that grow linearly with the number of tasks, limiting their scalability. More recent approaches investigate training a single LoRA by leveraging asymmetries in its parameterization; however, they often suffer from insufficient representational alignment across the LoRA parameter space. To address these limitations, we propose Stiefel Optimization and Aligned Rotation (SOAR), a two-stage framework for continual LoRA learning. First, we constrain the A matrix to lie on the Stiefel manifold, thereby reducing coordinate misalignment to the space of orthogonal rotation matrices. Second, we introduce an Orthogonal Procrustes alignment procedure for the B matrix to estimate the optimal rotation, coupled with a per-rank retention merging strategy that quantifies directional agreement between representations of new and previously learned tasks. We conduct extensive experiments across multiple models and benchmarks, demonstrating that SOAR achieves strong continual learning performance without incurring additional memory overhead.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑