学习方向，规范化增益：LoRA 的后训练归一化
Learn the Directions, Normalize the Gains: Post-Training Normalization for LoRA
- Singapore Management University（新加坡管理大学）
- National University of Singapore（新加坡国立大学）
- Peking University（北京大学）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
针对LoRA适配失衡问题，提出后训练归一化方法LoRA-Norm，通过频谱再平衡和核范数恢复重平衡增益，无需额外训练，提升特化与能力保持。
AI中文摘要：
虽然低秩适配（LoRA）能够实现高效的任务特化，但其学习到的更新可能会损害目标任务之外的能力。我们识别出“适配失衡”现象：少数奇异方向主导了训练更新，使得其性能对增益的分配方式敏感。我们认为，“学习在何处适配并不能确保适配增益得到良好平衡”。这促使我们提出 LoRA-Norm，一种后训练归一化方法，它在保留学习到的方向的同时重新平衡其增益。LoRA-Norm 结合了频谱再平衡（对奇异值进行固定的非线性变换）和核范数恢复（保持原始总频谱质量）。它不需要校准数据或额外训练，且不引入推理开销。在两种骨干网络和三个适配任务中，LoRA-Norm 提升了平均特化能力和能力保持，在两项指标上均优于所评估的事后频谱剪枝和梯度引导编辑配置。更强的功能均衡并未带来一致的额外增益，这表明平衡适配器增益与均衡其响应是两个不同的目标。
英文摘要：
While Low-Rank Adaptation (LoRA) enables efficient task specialization, its learned updates can compromise capabilities beyond the target task. We identify \textbf{adaptation imbalance}: a few singular directions dominate the trained update, leaving its performance sensitive to how gains are allocated. We argue that \textbf{learning where to adapt does not ensure that adaptation gains are well balanced}. This motivates \textbf{LoRA-Norm}, a post-training normalization method that retains learned directions while rebalancing their gains. LoRA-Norm combines spectral rebalancing, a fixed nonlinear transformation of singular values, with nuclear-norm restoration, which preserves the original total spectral mass. It requires no calibration data or additional training and introduces no inference overhead. Across two backbones and three adaptation tasks, LoRA-Norm improves average specialization and capability retention, outperforming the evaluated post-hoc spectral pruning and gradient-guided editing configurations on both measures. Stronger functional equalization brings no consistent additional gains, revealing that balancing adapter gains and equalizing their responses are distinct objectives.
