一个学习率不够：用于LoRA微调的自适应各向异性学习率

中文摘要 AI 辅助

低秩适配（LoRA）已成为大型语言模型参数高效微调的标准方法。大多数LoRA变体遵循统一学习率约定，对每个适配器的每个秩一分量应用单一全局学习率。我们表明，这种约定忽视了模块内部显著的异质性，其中LoRA适配器的秩一分量以高度不均匀的速率更新，低速度模块收敛到集中的奇异谱，未充分利用名义秩预算。为解决此问题，我们提出一种自适应各向异性学习率模型，为每个秩一分量分配其自身的有效学习率，该学习率根据训练时信号在线计算，并按模块进行均值归一化以保持全局学习率预算。AnLR-LoRA使用AdamW优化过程中可用的两个信号（即函数空间速度和Adam信噪比）实例化此模型，作为一种无额外可训练参数的轻量方案。在常识推理、自然语言生成和视觉指令微调基准上，AnLR-LoRA持续优于LoRA，同时鼓励更广泛地使用秩容量，其增益在广泛的全局学习率范围内保持稳健，并可干净地迁移到其他LoRA变体。

英文摘要

Low-rank adaptation (LoRA) has become the standard for parameter-efficient fine-tuning of large language models. Most LoRA variants follow a uniform-LR convention, applying a single global learning rate across every rank-one component of every adapter. We show that this convention overlooks substantial within-module heterogeneity, where the rank-one components of a LoRA adapter update at highly uneven rates and low-velocity modules converge to concentrated singular spectra that underutilize the nominal rank budget. To address this, we propose an adaptive anisotropic learning-rate model that assigns each rank-one component its own effective learning rate, computed online from training-time signals and mean-normalized per module to preserve the global LR budget. AnLR-LoRA instantiates this model with two signals available during AdamW optimization, namely function-space velocity and Adam SNR, as a lightweight scheme with no extra trainable parameters. Across commonsense reasoning, natural language generation and visual instruction-tuning benchmarks, AnLR-LoRA consistently improves over LoRA while encouraging broader use of rank capacity, with gains that remain robust across a wide range of global learning rates and transfer cleanly to other LoRA variants.

发表机构

• University of New South Wales（新南威尔士大学）

机构由 AI 辅助整理，请以论文原文为准。