LoRA-TSD：基于Muon风格更新的LoRA切空间谱下降算法
LoRA-TSD: Tangent-Space Spectral Descent for LoRA via Muon-Style Updates
浏览论文内容
中文总结 AI 辅助
针对LoRA独立训练忽略低秩权重变化几何的问题，本文提出LoRA-TSD优化器，通过切空间谱下降提升效率与性能，在多模型多基准测试中优于竞争方法并具鲁棒性。
中文摘要 AI 辅助
低秩适配（LoRA）是微调大模型的标准方法，但当它的两个因子独立训练时，更新过程会忽略其诱导的低秩权重变化的几何结构。本文提出LoRA-TSD，一种优化器，它将每一步LoRA更新视为固定秩矩阵流形的切向量，并在该切空间内执行Muon的谱范数最速下降步，通过LoRA参数化自带的回缩映射将结果转换回因子形式。该步骤避免了对完整权重矩阵的昂贵操作，其回缩操作比现有流形方法使用的截断SVD回缩快达2.8倍。我们证明，本文代理函数的Frobenius范数版本可恢复LoRA-Pro，且确定了切空间投影梯度（即流形的黎曼梯度）是LoRA训练的自然平稳性度量，仅需从因子梯度即可计算。基于该度量，我们首次为LoRA-Pro和LoRA-TSD提供了全局收敛保证，其收敛速率可使因子梯度范数趋于零。在使用Llama-3.2-1B、Llama-3.1-8B和Qwen3-32B的6个常识推理和自然语言推理基准测试中，LoRA-TSD的性能优于所有竞争的LoRA优化器，且对适配器秩保持鲁棒性。代码可在指定URL获取。
英文摘要
Low-rank adaptation (LoRA) is the standard way to fine-tune large models, yet when its two factors are trained independently, the update ignores the geometry of the low-rank weight change it induces. We introduce LoRA-TSD, an optimizer that treats every LoRA step as a tangent vector of the fixed-rank matrix manifold and takes the spectral-norm steepest-descent step of Muon inside that tangent space, mapping the result back to the factors through a retraction native to the LoRA parametrization. The step avoids expensive operations on full weight matrices, and its retraction is up to $2.8\times$ cheaper than the truncated-SVD retraction used by prior manifold methods. We prove that the Frobenius-norm version of our surrogate recovers LoRA-Pro, and we identify the tangent-projected gradient, the Riemannian gradient of the manifold, as the stationarity measure natural to LoRA training and computable from the factor gradients alone. Under this measure we give the first global convergence guarantees for both LoRA-Pro and LoRA-TSD, with rates that drive the factor-gradient norms to zero. Across six commonsense and natural-language-inference benchmarks with Llama-3.2-1B, Llama-3.1-8B and Qwen3-32B, LoRA-TSD outperforms every competing LoRA optimizer and stays robust to the adapter rank. Code is available at https://github.com/brain-lab-research/LoRA-TSD.
发表机构
- Basic Research of Artificial Intelligence Laboratory (BRAIn Lab)（人工智能基础研究实验室（BRAIn实验室））
- SB AI Lab（SB人工智能实验室）
- Innopolis University（Innopolis大学）
机构由 AI 辅助整理，请以论文原文为准。
