DoRAN: 通过噪声注入和辅助网络稳定化权重分解低秩适应
DoRAN: Stabilizing Weight-Decomposed Low-Rank Adaptation via Noise Injection and Auxiliary Networks
AI总结：
DoRAN通过噪声注入和辅助网络提升低秩适应的稳定性与样本效率
摘要：
参数高效微调（PEFT）方法已成为适应大规模模型的标准范式。在这些技术中，权重分解低秩适应（DoRA）通过显式将预训练权重分解为幅度和方向成分，已被证明能提高低秩适应（LoRA）方法的学习能力和训练稳定性。在本工作中，我们提出DoRAN，一种新的技术，旨在稳定训练并提升DoRA的样本效率。我们的框架引入了两个关键组件：（i）将可学习噪声注入到DoRA权重分解的分母中，作为自适应正则化器以缓解不稳定性并提高低秩矩阵的估计率；（ii）用静态低秩矩阵替换为辅助网络，使它们动态生成，从而在查询和值投影矩阵之间实现参数耦合，理论和实验证明了样本效率的提升。在视觉和语言基准上的全面实验表明，DoRAN在性能上始终优于LoRA、DoRA和其他PEFT基线，证明了结合基于噪声的正则化与基于网络的参数生成的有效性。
英文摘要：
Parameter-efficient fine-tuning (PEFT) methods have become the standard paradigm for adapting large-scale models. Among these techniques, Weight-Decomposed Low-Rank Adaptation (DoRA) has been shown to improve both the learning capacity and training stability of the Low-Rank Adaptation (LoRA) method by explicitly decomposing pre-trained weights into magnitude and directional components. In this work, we propose DoRAN, a new technique designed to stabilize training and boost the sample efficiency of DoRA. Our framework introduces two key components: (i) the injection of learnable noise into the denominator of DoRA weight decomposition, which serves as an adaptive regularizer to mitigate instabilities and improve the estimation rate of low-rank matrices; and (ii) the replacement of static low-rank matrices with auxiliary networks that generate them dynamically, enabling parameter coupling between the query and value projection matrices, leading to improved sample efficiency both theoretically and empirically. Comprehensive experiments on vision and language benchmarks show that DoRAN consistently outperforms LoRA, DoRA, and other PEFT baselines, underscoring the effectiveness of combining noise-based regularization with network-based parameter generation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
