Bayesian-LoRA:通过可微贝叶斯门使用最优量化级别和秩值的基于 LoRA 的参数高效微调
Bayesian-LoRA: LoRA based Parameter Efficient Fine-Tuning using Optimal Quantization levels and Rank Values trough Differentiable Bayesian Gates
AI总结:
提出 Bayesian-LoRA,利用可微贝叶斯门为每个 LoRA 矩阵联合选择最优秩与量化级别,在 GLUE 上保持或提升性能并减少约 70% 位运算。
摘要:
在自然语言处理中,通常的做法是在通用领域上预训练单个模型,然后针对下游任务对其进行微调。然而,对于大型语言模型而言,微调整个模型在计算上可能非常昂贵,从而导致极高的能耗。因此,近来研究人员提出了若干参数高效微调(PEFT)方法。其中最流行的方法之一是低秩适配(LoRA),其核心思想是将预训练模型的更新权重分解为两个低秩矩阵。但是,已有方法要么在所有不同权重矩阵上使用相同的秩值,而这已被证明是一种次优选择;要么完全不使用任何量化技术,而量化正是影响模型能耗的最重要因素之一。在本研究中,我们提出 Bayesian-LoRA,它从贝叶斯角度处理低秩适配和量化,通过对量化级别和秩值同时采用先验分布来实现。因此,B-LoRA 能够针对特定下游任务微调预训练模型,并为每个低秩矩阵找到最优秩值和量化级别。我们通过在 GLUE 基准上微调预训练的 DeBERTaV3 来验证所提出的模型。此外,我们将其与相关基线进行比较,并给出定性和定量结果,展示所提方法如何学习最优秩的量化矩阵。B-LoRA 的性能与基线相当或更好,同时与基线方法相比将位运算总数减少约 70%。
英文摘要:
It is a common practice in natural language processing to pre-train a single model on a general domain and then fine-tune it for downstream tasks. However, when it comes to Large Language Models, fine-tuning the entire model can be computationally expensive, resulting in very intensive energy consumption. As a result, several Parameter Efficient Fine-Tuning (PEFT) approaches were recently proposed. One of the most popular approaches is low-rank adaptation (LoRA), where the key insight is decomposing the update weights of the pre-trained model into two low-rank matrices. However, the proposed approaches either use the same rank value across all different weight matrices, which has been shown to be a sub-optimal choice, or do not use any quantization technique, one of the most important factors when it comes to a model's energy consumption. In this work, we propose Bayesian-LoRA which approaches low-rank adaptation and quantization from a Bayesian perspective by employing a prior distribution on both quantization levels and rank values. As a result, B-LoRA is able to fine-tune a pre-trained model on a specific downstream task, finding the optimal rank values and quantization levels for every low-rank matrix. We validate the proposed model by fine-tuning a pre-trained DeBERTaV3 on the GLUE benchmark. Moreover, we compare it to relevant baselines and present both qualitative and quantitative results, showing how the proposed approach is able to learn optimal-rank quantized matrices. B-LoRA performs on par with or better than the baselines while reducing the total number of bit operations by roughly 70% compared to the baseline methods.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。