arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.00084cs.LG
收藏

为何LoRA具有抗标签噪声性:一种用于噪声鲁棒性参数高效微调的理论框架

Why LoRA Resists Label Noise: A Theoretical Framework for Noise-Robust Parameter-Efficient Fine-Tuning

Georgia Institute of Technology, Atlanta, GA, USA(佐治亚理工学院)

Brady Steele

AI总结:

本文提出了一种理论框架,解释LoRA对标签噪声的抗性,并提出RACT方法通过秩差异实现噪声检测,实验验证了其有效性。

摘要:

参数高效微调方法,如低秩适应(LoRA),已成为适应大型预训练模型的主导范式。我们提出了一种理论框架,解释了一个未被充分探索的性质:LoRA固有的抗标签噪声性。我们的分析揭示了三个关键见解。首先,我们证明了秩为r的LoRA在样本量超过O(r(d+k-r))时无法记忆所有可能的标签分配,限制了其拟合任意噪声的能力。其次,我们推导了最优秩平衡近似偏差和噪声诱导方差,表明其随着噪声率降低而减少。第三,我们建立了时间分离:干净的模式在早期学习,而噪声记忆发生在后期。我们提出RACT(秩感知课程训练),利用秩差异进行噪声检测。实验验证了我们的预测,RACT在AG News上实现91.1%的F1值用于噪声检测,同时保持91.46%的准确率,与缺乏噪声检测能力的基线方法相竞争。

英文摘要:

Parameter-efficient fine-tuning methods like Low-Rank Adaptation (LoRA) have become the dominant paradigm for adapting large pretrained models. We present a theoretical framework explaining an underexplored property: LoRA's inherent resistance to label noise. Our analysis reveals three key insights. First, we prove that rank-$r$ LoRA cannot memorize all possible label assignments once the sample size exceeds $O(r(d+k-r))$, limiting its capacity to fit arbitrary noise. Second, we derive an optimal rank balancing approximation bias and noise-induced variance, showing it decreases with noise rate. Third, we establish temporal separation: clean patterns are learned early while noise memorization occurs later. We propose RACT (Rank-Aware Curriculum Training), leveraging rank discrepancy for noise detection. Experiments validate our predictions, with RACT achieving 91.1% F1 for noise detection on AG News while maintaining 91.46% accuracy, competitive with baselines that lack noise detection capability.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑