arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2611.04239cs.LGcs.AI
收藏

LIRA:一种用于低秩适配的更具可解释性的动态损失感知秩分配方法

LIRA: A More Interpretable Dynamic Loss-Informed Rank Allocation for Low-Rank Adaptation

South China University of Technology(华南理工大学)

Sihao Wang, Bozheng Li, Xuemiao Xu, Haoxin Yang, Yujie Xu, Jie Tang

AI总结:

针对LoRA秩分配可解释性不足的问题,提出动态秩LoRA方法LIRA,通过二阶近似评分结合DELTA算法实现可解释秩分配,在RoBERTa-base GLUE任务中优于基线且估计成本更低。

摘要:

低秩适配(LoRA)已成为一种广泛应用于大型预训练模型的参数高效微调方法。然而,标准LoRA通常采用固定的秩配置,而不同层和不同阶段的适配需求存在差异;近期的自适应秩变体往往依赖启发式规则或一阶信号,这些信号仅能间接关联秩重新分配与局部损失行为,导致每次秩重新分配的依据缺乏足够可解释性。针对这一缺口,我们提出LIRA(损失感知秩分配),一种旨在提升在线秩重新分配可解释性的动态秩LoRA方法。LIRA通过二阶近似,基于各层近期LoRA因子移动对应的局部损失扰动估计幅度对其进行评分,使得每次秩变化的依据可逐项解释。LIRA会定期应用DELTA(位移评估损失轨迹分配)——一种基于位移的LoRA秩重新分配算法,在匹配预算下生成离散有效秩,同时兼容标准(A,B)适配器。在La-LoRA/LoRA协议下的RoBERTa-base GLUE任务中,LIRA优于该协议下报告的均匀LoRA和自适应秩基线;轨迹指数移动平均(Trajectory EMA)在更低估计成本下达到与定点费舍尔(Fixed-point Fisher)相当的性能。

英文摘要:

Low-rank adaptation (LoRA) has become a widely used parameter-efficient fine-tuning method for large pretrained models. However, standard LoRA typically assigns a fixed rank profile despite layer- and stage-dependent adaptation needs, while recent adaptive-rank variants often rely on heuristic or first-order signals that provide only an indirect link between rank reallocation and local loss behavior. This leaves the rationale behind each rank reallocation insufficiently interpretable. Motivated by this gap, we propose LIRA (Loss-Informed Rank Allocation), a dynamic-rank LoRA method designed to make online rank reallocation more interpretable. LIRA scores each layer by the estimated magnitude of the local loss perturbation associated with its recent LoRA-factor movements under a second-order approximation, making the basis for each rank change interpretable term by term. By periodically applying DELTA (Displacement-Evaluated Loss-Trajectory Allocation), a displacement-based algorithm for LoRA rank reallocation, LIRA produces discrete effective ranks under matched budgets while remaining compatible with standard (A,B) adapters. On RoBERTa-base GLUE under the La-LoRA/LoRA protocol, LIRA outperforms the uniform-LoRA and adaptive-rank baselines reported under this protocol; Trajectory EMA attains performance competitive with Fixed-point Fisher at lower estimation cost.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑