发表机构
National University of Singapore; Huawei; The University of Hong Kong; Hong Kong Baptist University（新加坡国立大学; 华为; 香港大学; 香港浸会大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文提出基于lp正则化的自动秩分配方法，通过正则化秩一组件能量并推导隐式阈值准则，在自然语言理解与问答任务上达到与现有LoRA基线相当的性能。
AI 中文摘要
低秩自适应（LoRA）已成为大语言模型中一种流行的参数高效微调方法。LoRA的一个关键挑战是如何确定每个自适应矩阵的秩，因为秩直接控制其容量和效率。现有的自适应秩方法通常根据手动设计的重要性分数来分配秩，这些分数并非直接来源于优化目标。在本工作中，我们提出了基于lp正则化（其中0<p<1）的$\u2113_p$-LoRA方法，这是一种在信号处理和统计学中经典的稀疏诱导技术。具体而言，我们对每个秩一LoRA组件的能量进行正则化，鼓励冗余组件消失，同时保留重要组件。我们推导了相应的近端子问题，并将矩阵优化简化为二维问题，从而得出一个用于识别冗余组件的隐式阈值准则。在自然语言理解和问答任务上的实验表明，所提出的方法达到了与现有LoRA基线相当的性能。
英文摘要
Low-rank adaptation (LoRA) has become a popular parameter-efficient fine-tuning method for large language models. A key challenge in LoRA is how to determine the rank of each adaptation matrix, as rank directly controls its capacity and efficiency. Existing adaptive-rank methods typically allocate ranks according to manually designed importance scores, which are not directly derived from an optimization objective. In this work, we propose $\ell_p$-LoRA, a principled rank-allocation method based on $\ell_p$ regularization with $0<p<1$, which is a classical sparsity-inducing technique in signal processing and statistics. Specifically, we regularize the energy of each rank-one LoRA component, encouraging redundant components to vanish while preserving important ones. We derive the corresponding proximal subproblem and reduce the matrix optimization to a two-dimensional problem, leading to an implicit thresholding criterion for identifying redundant components. Experiments on natural language understanding and question-answering tasks demonstrate that the proposed method achieves competitive performance with existing LoRA baselines.
Comments4 pages of main text and 1 page of reference
