LAARA：用于参数高效微调的层感知自适应秩分配
LAARA: Layer-Aware Adaptive Rank Allocation for Parameter-Efficient Fine-Tuning
浏览论文内容
中文总结 AI 辅助
研究针对低秩自适应中各层适应需求不同但现有方法统一分配适配器秩的问题，提出LAARA框架，利用轻量级对角Fisher估计动态分配秩，结合多种机制，实验表明其在少参数情况下性能优于现有方法，为自适应参数高效微调提供基础。
中文摘要 AI 辅助
低秩自适应被广泛用于参数高效微调，但现有方法通常为每个变压器层分配相同的适配器秩，尽管它们的适应需求不同。本文通过理论和实验表明统一秩分配本质上是次优的。基于此，我们提出了LAARA（层感知自适应秩分配框架），这是一个无搜索框架，利用训练期间计算的轻量级对角Fisher估计动态分配秩。LAARA结合逐投影归一化、对数压缩、混合适配器重要性估计和投票改变阻尼机制，以产生稳定高效的秩适应。在GLUE和MathInstruct基准上的实验表明，LAARA在使用显著更少可训练参数的情况下，始终匹配或优于LoRA、AdaLoRA、DyLoRA和Bitfit等流行的现有方法。我们的结果表明，Fisher引导的秩分配为自适应参数高效微调提供了一个有原则且有效的基础。代码可在指定链接公开获取。
英文摘要
Low-Rank Adaptation is widely used for parameter-efficient fine-tuning, yet existing methods typically assign the same adapter rank to every transformer layer despite their heterogeneous adaptation requirements. In this work, we show theoretically and empirically that uniform rank allocation is fundamentally suboptimal. Motivated by this observation, we propose LAARA (Layer Aware Adaptive Rank Allocation framework), a search-free framework that dynamically allocates ranks using lightweight diagonal Fisher estimates computed during training. LAARA combines projection-wise normalization, logarithmic compression, blended adapter importance estimation, and a vote-to-change dampening mechanism to produce stable and efficient rank adaptation. Experiments on GLUE and MathInstruct benchmark demonstrate that LAARA consistently matches or outperforms popular state of the art approaches such as LoRA, AdaLoRA, DyLoRA, and Bitfit while using significantly fewer trainable parameters. Our results show that Fisher-guided rank allocation provides a principled and effective foundation for adaptive parameter-efficient fine-tuning. The code is publicly available at: https://anonymous.4open.science/r/LAARA-D305/LAARA.py
发表机构
- Indian Institute of Technology, Patna（印度理工学院巴特那分校）
机构由 AI 辅助整理，请以论文原文为准。
