arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.10212cs.LG
收藏

LoRA的精度与秩的权衡:梯度流分析

Rank-Accuracy Trade-off for LoRA: A Gradient-Flow Analysis

Department of Engineering Sciences and Applied Mathematics, Northwestern University, Evanston, USA(工程科学与应用数学系,西北大学,埃文斯顿,美国);Department of Industrial Engineering and Management Sciences, Northwestern University, Evanston, USA(工业工程与管理科学系,西北大学,埃文斯顿,美国)

Michael Rushka, Diego Klabjan

AI总结:

本文通过梯度流分析,揭示了LoRA精度与更新秩之间的关系,建立了两种损失函数下秩与精度的闭式关系。

摘要:

先前的经验研究表明,LoRA在下游微调任务上实现的精度与全参数方法相当,即使在秩为1的更新情况下也是如此。相比之下,LoRA的精度与更新秩之间的理论基础仍然相对未被探索。在本文中,我们从动态系统角度比较了秩为r的LoRA更新与全参数更新在微调任务上的精度。我们分别在全秩和低秩情况下进行梯度流分析,以建立LoRA的秩与精度之间在两个损失函数下的显式关系。虽然先前的工作中已经提出了LoRA的梯度流方程,但本文严格推导了其形式,并证明了同时和顺序LoRA参数更新的梯度流方程是相同的。然后,我们利用由此得到的动态系统方程,获得LoRA秩与精度之间在迹平方和Frobenius范数低秩近似损失函数下的闭式关系。

英文摘要:

Previous empirical studies have shown that LoRA achieves accuracy comparable to full-parameter methods on downstream fine-tuning tasks, even for rank-1 updates. By contrast, the theoretical underpinnings of the dependence of LoRA's accuracy on update rank remain relatively unexplored. In this work, we compare the accuracy of rank-r LoRA updates against full-parameter updates for fine-tuning tasks from a dynamical systems perspective. We perform gradient flow analysis in both full-rank and low-rank regimes to establish explicit relationships between rank and accuracy for two loss functions under LoRA. While gradient flow equations for LoRA are presented in prior work, we rigorously derive their form and show that they are identical for simultaneous and sequential LoRA parameter updates. We then use the resulting dynamical system equations to obtain closed-form relationships between LoRA rank and accuracy for trace-squared and Frobenius-norm low-rank approximation loss functions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑