arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2502.09376cs.LG
收藏

LoRA训练可证明收敛到低秩全局最小值,或显著失败(但很可能不会失败)

LoRA Training Provably Converges to a Low-Rank Global Minimum or It Fails Loudly (But it Probably Won't Fail)

Harvard University(哈佛大学)

Junsu Kim, Jaeyeon Kim, Ernest K. Ryu

AI总结:

本文在无线性化假设下分析LoRA损失景观,证明在通用状态下训练要么收敛到低秩小幅值全局最小值,要么收敛到高秩大幅值解,并揭示零初始化与权重衰减诱导的隐式偏置使LoRA通常成功找到全局最小值。

摘要:

低秩适应(LoRA)已成为微调大型基础模型的标准方法。然而,我们对LoRA的理论理解仍然有限,因为先前对LoRA训练动力学的分析要么依赖线性化论证,要么考虑高度简化的设置。在本工作中,我们在没有此类限制性假设的情况下分析LoRA损失景观。我们定义两种状态:一种“特殊状态”,包括线性化论证成立的理想化设置;另一种“通用状态”,代表线性化论证不成立的更现实设置。在通用状态下,我们证明LoRA训练要么收敛到具有低秩和小幅值的全局最小解,要么收敛到具有高秩和大幅值的性质不同的解。最后,我们论证LoRA训练中的零初始化和权重衰减会诱导一种隐式偏置,使其偏向参数空间中低秩、小幅值区域——全局最小值所在之处——从而阐明为什么LoRA训练通常能够成功找到全局最小值。

英文摘要:

Low-rank adaptation (LoRA) has become a standard approach for fine-tuning large foundation models. However, our theoretical understanding of LoRA remains limited as prior analyses of LoRA's training dynamics either rely on linearization arguments or consider highly simplified setups. In this work, we analyze the LoRA loss landscape without such restrictive assumptions. We define two regimes: a "special regime", which includes idealized setups where linearization arguments hold, and a "generic regime" representing more realistic setups where linearization arguments do not hold. In the generic regime, we show that LoRA training converges to a global minimizer with low rank and small magnitude, or a qualitatively distinct solution with high rank and large magnitude. Finally, we argue that the zero-initialization and weight decay in LoRA training induce an implicit bias toward the low-rank, small-magnitude region of the parameter space -- where global minima lie -- thus shedding light on why LoRA training usually succeeds in finding global minima.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑