arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.14713cs.LGcs.CL
收藏

QuAILoRA:面向LoRA的量化感知初始化

QuAILoRA: Quantization-Aware Initialization for LoRA

Amazon(亚马逊公司)

Neal Lawton, Aishwarya Padmakumar, Judith Gaspers, Jack FitzGerald, Anoop Kumar, Greg Ver Steeg, Aram Galstyan

AI总结:

本文提出QuAILoRA,一种面向LoRA的量化感知初始化方法,通过在初始化阶段减少量化误差来缓解QLoRA微调中的性能损失,在几乎不增加内存成本的情况下显著提升4位量化模型的验证困惑度和下游任务准确率。

摘要:

QLoRA通过量化基础大语言模型(LLM)来降低使用LoRA微调大语言模型的内存成本。然而,量化会引入量化误差,从而对微调后的模型性能产生负面影响。在本文中,我们提出QuAILoRA,一种面向LoRA的量化感知初始化方法,通过在初始化阶段减少量化误差来缓解这一负面影响。我们的方法仅需少量计算开销来计算这种量化感知初始化,且不会增加微调的内存成本。我们使用多种不同模型规模和模型系列,在多个因果语言建模和下游评估任务上评估了该方法。我们观察到,几乎所有使用QuAILoRA微调的LLM都取得了更好的验证困惑度。在下游任务评估中,我们发现QuAILoRA带来的改进与量化误差的负面影响成比例。平均而言,将QuAILoRA应用于4位QLoRA模型,可获得将量化精度加倍至8位时所取得的验证困惑度下降幅度的75%和下游任务准确率提升幅度的86%,同时不增加微调期间的GPU内存占用。

英文摘要:

QLoRA reduces the memory-cost of fine-tuning a large language model (LLM) with LoRA by quantizing the base LLM. However, quantization introduces quantization errors that negatively impact model performance after fine-tuning. In this paper we introduce QuAILoRA, a quantization-aware initialization for LoRA that mitigates this negative impact by decreasing quantization errors at initialization. Our method spends a small amount of computational overhead to compute this quantization-aware initialization, without increasing the memory-cost of fine-tuning. We evaluate our method on several causal language modeling and downstream evaluation tasks using several different model sizes and families. We observe that almost all LLMs fined-tuned with QuAILoRA achieve better validation perplexity. When evaluated on downstream tasks, we find that QuAILoRA yields improvements proportional to the negative effect of quantization error. On average, applying QuAILoRA to 4-bit QLoRA models yields 75% of the validation perplexity decrease and 86% of the downstream task accuracy increase as doubling the quantization precision to 8-bit, without increasing GPU memory utilization during fine-tuning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑