低秩适应中秩分配的统计推断
Statistical Inference for Rank Allocation in Low-Rank Adaptation
浏览论文内容
中文总结 AI 辅助
研究低秩适应中在固定参数预算下分配秩资源的问题，提出StatLoRA方法，将其表述为统计假设检验问题，通过检验统计量和p值确定组件取舍，实验表明该方法在匹配秩预算下性能良好，支持了分配规则稳定性及渐近理论。
中文摘要 AI 辅助
低秩适应（LoRA）已成为大语言模型广泛使用的参数高效微调方法。由于不同模块和层对下游适应的贡献可能不同，在固定参数预算下分配秩资源对于平衡效率、表现力和泛化能力是一个重要问题。现有自适应秩方法主要通过精心设计的重要性分数来解决此问题，缺乏明确的统计解释。本文将LoRA秩分配表述为统计假设检验问题，并提出基于统计推断的秩分配方法StatLoRA。StatLoRA将每个LoRA组件与一个检验统计量相关联，并使用估计的p值来确定在规定的秩预算下哪些组件应保留或修剪。我们的中心极限理论支持所提出的测试过程。我们在自然语言理解、自然语言生成和问答任务中对DeBERTaV3-base、BART-Large和Qwen2.5-7B进行LoRA微调评估StatLoRA。实验表明，在匹配的秩预算下，StatLoRA比普通LoRA、AdaLoRA和IGU-LoRA具有可比或更好的性能。敏感性分析和实证诊断进一步支持了基于假设检验的分配规则的稳定性，并为组件分数的渐近理论提供了经验证据。
英文摘要
Low-rank adaptation (LoRA) has become a widely used parameter-efficient fine-tuning method for large language models. Since different modules and layers may contribute unequally to downstream adaptation, allocating rank resources under a fixed parameter budget is an important problem for balancing efficiency, expressiveness, and generalization. Existing adaptive rank methods address this problem mainly through carefully designed importance scores constructed from gradient-derived sensitivity and uncertainty measures, without an explicit statistical interpretation. In this paper, we formulate LoRA rank allocation as a statistical hypothesis testing problem and propose StatLoRA, a statistical inference-based rank allocation method. StatLoRA associates each LoRA component with a test statistic and uses estimated p-values to determine which components should be retained or pruned under a prescribed rank budget. The proposed testing procedure is supported by our central limit theory for stochastic optimizer trajectories. In particular, we establish asymptotic normality for a broad class of commonly used optimizers in deep learning, including AdamW, and derive the corresponding asymptotic distributions for the proposed component scores used in hypothesis testing. We evaluate StatLoRA on LoRA fine-tuning of DeBERTaV3-base, BART-Large, and Qwen2.5-7B across natural language understanding, natural language generation, and question answering tasks. Experiments show that StatLoRA achieves comparable or better performance than vanilla LoRA, AdaLoRA, and IGU-LoRA under matched rank budgets. Sensitivity analyses and empirical diagnostics further support the stability of the proposed hypothesis-testing-based allocation rule and provide empirical evidence for the asymptotic theory of component scores.
发表机构
- Department of Mathematics, National University of Singapore（数学系，新加坡国立大学）
- Department of Mathematics and Department of Electrical and Computer Engineering, National University of Singapore（数学系和电气与计算机工程系，新加坡国立大学）
机构由 AI 辅助整理，请以论文原文为准。
