LoSiA：通过子网定位与优化实现高效高秩微调
LoSiA: Efficient High-Rank Fine-Tuning via Subnet Localization and Optimization
AI总结：
LoSiA通过梯度稀疏性分析定位并优化关键子网络，实现高效高秩微调，减少矩阵乘法，性能接近全量微调且训练时间最短，并减少遗忘。
摘要：
参数高效微调（PEFT）方法，如LoRA，通过引入低秩分解矩阵显著减少了可训练参数的数量。然而，现有方法在领域专业化任务中执行大量的矩阵乘法，导致计算效率低下和微调性能次优。因此，我们提出了LoSiA（低资源子网集成适配），一种创新方法，在训练过程中动态定位并优化关键参数。具体而言，它利用梯度稀疏性分析识别一个子网络，并将其作为可训练目标进行优化。这种设计通过仅更新子网络参数实现有效的高秩适配，减少了额外的矩阵乘法。我们还提出了LoSiA-Pro，一种更快的LoSiA实现，与LoRA相比，训练延迟降低了约27%。大量评估表明，与全量微调相比，我们的方法性能下降极小，同时在领域专业化和常识推理任务中所需训练时间最少。进一步分析显示，LoSiA在持续训练中也减少了遗忘。源代码可在https://github.com/KlozeWang/LoSiA获取。
英文摘要：
Parameter-Efficient Fine-Tuning (PEFT) methods, such as LoRA, significantly reduce the number of trainable parameters by introducing low-rank decomposition matrices. However, existing methods perform extensive matrix multiplications in domain specialization tasks, resulting in computational inefficiency and sub-optimal fine-tuning performance. Hence, we propose LoSiA(Low-Resources Subnet Integration Adaptation), an innovative method that dynamically localizes and optimizes critical parameters during the training process. Specifically, it identifies a sub-network using gradient sparsity analysis and optimizes it as the trainable target. This design enables effective high-rank adaptation by updating only the sub-network parameters, reducing the additional matrix multiplication. We also present LoSiA-Pro, a faster implementation of LoSiA, which reduces the training latency by about $27\%$ compared to LoRA. Extensive evaluations show that our method achieves minimal performance drop compared to full fine-tuning, while requiring the least training time across domain specialization and common-sense reasoning tasks. Further analysis shows that LoSiA also reduces forgetting during continued training. The source code is available at https://github.com/KlozeWang/LoSiA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
