arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2607.02576stat.MLcs.LG
收藏

CORA:基于切片的相干正交旋转,用于基于SVD的低秩适配

CORA: Per-Slice Coherent Orthogonal Rotation for SVD-based Low-Rank Adaptation

Purdue University(普渡大学);Shanghai Institute for Mathematics and Interdisciplinary Sciences / Fudan University(上海数学与交叉科学研究院 / 复旦大学);Futurewei Technologies, Inc.(未来科技有限公司)

Pengcheng Wang, Ziran Liu, Wei Wang, Wei Jiang

AI总结:

针对现有基于SVD的PEFT方法未保留预训练奇异基耦合几何的问题,提出逐切片相干正交旋转适配方法CORA,参数量远低于同类方法,在常识推理、代码生成任务上性能更优。

摘要:

参数高效微调(PEFT)通常通过低秩更新适配预训练权重,近期方法进一步利用基础权重的奇异值分解(SVD)完成初始化或子空间选择。但这类方法未显式保留预训练左、右奇异基之间的耦合几何关系。受最小扰动理论(表明稳定微调遵循相干SVD旋转,即单个正交矩阵$Q$同时作用于左奇异基$U_0$和右奇异基$V_0$)启发,本文证明了逐切片的类似结论:$W_0$的每个行切片可通过共享正交旋转$Q_i$同时作用于其左基$U_i$和右基$V_i$,结合对角谱偏移完成适配。本文将该范式实现为CORA(相干正交旋转适配),对$W_0$的秩-$r$ SVD截断结果施加逐切片正交旋转与逐层对角缩放。CORA的每个线性层仅含$\tfrac{1}{2}m(r-1)$个可训练参数,相同秩下比LoRA少约4倍参数量。在参数量少约8倍的前提下,CORA在常识推理与代码生成任务上的表现优于LoRA、DoRA、PiSSA和MiLoRA。

英文摘要:

Parameter-Efficient Fine-Tuning (PEFT) commonly adapts pretrained weights through low-rank updates, and recent methods further exploit the singular value decomposition (SVD) of the base weight for initialization or subspace selection. However, these methods do not explicitly preserve the coupled geometry between the pretrained left and right singular bases. Motivated by recent minimum-perturbation theory, which shows that stable finetuning follows a coherent SVD rotation in which a single orthogonal $Q$ acts on both the left singular basis $U_0$ and the right singular basis $V_0$, we prove a per-slice analogue: each row slice of $W_0$ can be adapted by a shared orthogonal rotation $Q_i$ on its left basis $U_i$ and right basis $V_i$ together with a diagonal spectrum shift. We implement this form as CORA (Coherent Orthogonal Rotation Adaptation), which applies per-slice orthogonal rotations and a per-layer diagonal scale to the rank-$r$ SVD truncation of $W_0$. CORA uses $\tfrac{1}{2}m(r{-}1)$ trainable parameters per linear layer, about $4{\times}$ fewer than LoRA at the same rank. CORA outperforms LoRA, DoRA, PiSSA, and MiLoRA on commonsense reasoning and code generation while using about $8{\times}$ fewer parameters.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑