IGU-LoRA: 通过集成梯度和不确定性感知评分实现自适应秩分配
IGU-LoRA: Adaptive Rank Allocation via Integrated Gradients and Uncertainty-Aware Scoring
AI总结:
IGU-LoRA通过计算层内集成梯度敏感度并结合不确定性感知方案,实现自适应秩分配,提升参数高效微调效果。
摘要:
随着大语言模型(LLMs)参数规模扩大,全参数微调变得计算和内存不可行。参数高效微调(PEFT)通过仅更新少量任务特定参数来缓解这一问题。尽管低秩适应(LoRA)被广泛采用,但其强制所有层使用相同秩,尽管层重要性存在显著差异。最近的自适应秩变种(如AdaLoRA)根据重要性评分分配秩,但通常依赖于瞬时梯度,仅捕捉局部敏感性,忽略了同一层内的非局部路径效应,导致评分不稳定和偏倚。为了解决这一限制,我们引入了IGU-LoRA,一种自适应秩LoRA,其(i)计算层内集成梯度(IG)敏感度并将它们聚合为层级分数用于秩分配,(ii)应用基于指数移动平均和偏差跟踪的不确定性感知方案以抑制噪声更新并校准秩选择。理论上,我们证明了在路径Hessian-Lipschitz条件下,参数空间IG的复合梯形法则近似误差上界,这指导了四则预算。在多样任务和架构上,IGU-LoRA在匹配的参数预算下始终优于强PEFT基线,提高了下游准确性和鲁棒性。消融实验确认了路径内层敏感度估计和不确定性感知选择对有效秩分配的贡献。我们的代码可在https://github.com/withyou12/igulora.git上公开获取。
英文摘要:
As large language models (LLMs) scale to billions of parameters, full-parameter fine-tuning becomes compute- and memory-prohibitive. Parameter-efficient fine-tuning (PEFT) mitigates this issue by updating only a small set of task-specific parameters while keeping the base model frozen. Among PEFT approaches, low-rank adaptation (LoRA) is widely adopted; however, it enforces a uniform rank across layers despite substantial variation in layer importance, motivating {layerwise} rank allocation. Recent adaptive-rank variants (e.g., AdaLoRA) allocate ranks based on importance scores, yet typically rely on instantaneous gradients that capture only local sensitivity, overlooking non-local, pathwise effects within the same layer, which yields unstable and biased scores. To address this limitation, we introduce IGU-LoRA, an adaptive-rank LoRA that (i) computes within-layer Integrated Gradients (IG) sensitivities and aggregates them into a layer-level score for rank allocation, and (ii) applies an uncertainty-aware scheme using exponential moving averages with deviation tracking to suppress noisy updates and calibrate rank selection. Theoretically, we prove an upper bound on the composite trapezoidal rule approximation error for parameter-space IG under a pathwise Hessian-Lipschitz condition, which informs the quadrature budget. Across diverse tasks and architectures, IGU-LoRA consistently outperforms strong PEFT baselines at matched parameter budgets, improving downstream accuracy and robustness. Ablations confirm the contributions of pathwise within-layer sensitivity estimates and uncertainty-aware selection to effective rank allocation. Our code is publicly available at https://github.com/withyou12/igulora.git
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。