Delta-CoMe: 面向大语言模型的无训练混合精度Delta压缩
Delta-CoMe: Training-Free Delta-Compression with Mixed-Precision for Large Language Models
AI总结:
针对现有delta权重压缩方法损害任务特定微调大语言模型性能的问题,提出基于奇异值长尾分布的混合精度delta量化方法,在多类模型上性能接近全微调模型且大幅超越基线,兼容多种骨干模型具通用性。
摘要:
微调是使大语言模型(LLMs)适应多样化应用的关键过程。在某些场景下,例如多租户服务,部署多个LLM以满足复杂需求变得必要。近期研究建议将微调后的LLM分解为基础模型和对应的delta权重,然后使用低秩或低位方法压缩这些delta权重以降低成本。在本工作中,我们观察到现有低秩和低位压缩方法会显著损害任务特定微调LLM(例如用于数学问题的WizardMath)的性能。受delta权重中奇异值长尾分布的启发,我们提出一种使用混合精度的delta量化方法。该方法对对应较大奇异值的奇异向量采用更高位表示。我们在多种微调LLM上评估了该方法,包括数学LLM、代码LLM、聊天LLM甚至视觉语言模型(VLMs)。实验结果表明,我们的方法性能与全微调LLM相当,大幅超越低秩和低位基线。此外,我们证明该方法与Llama-2、Llama-3和Mistral等多种骨干LLM兼容,突显了其通用性。
英文摘要:
Fine-tuning is a crucial process for adapting large language models (LLMs) to diverse applications. In certain scenarios, such as multi-tenant serving, deploying multiple LLMs becomes necessary to meet complex demands. Recent studies suggest decomposing a fine-tuned LLM into a base model and corresponding delta weights, which are then compressed using low-rank or low-bit approaches to reduce costs. In this work, we observe that existing low-rank and low-bit compression methods can significantly harm the model performance for task-specific fine-tuned LLMs (e.g., WizardMath for math problems). Motivated by the long-tail distribution of singular values in the delta weights, we propose a delta quantization approach using mixed-precision. This method employs higher-bit representation for singular vectors corresponding to larger singular values. We evaluate our approach on various fine-tuned LLMs, including math LLMs, code LLMs, chat LLMs, and even VLMs. Experimental results demonstrate that our approach performs comparably to full fine-tuned LLMs, surpassing both low-rank and low-bit baselines by a considerable margin. Additionally, we show that our method is compatible with various backbone LLMs, such as Llama-2, Llama-3, and Mistral, highlighting its generalizability.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。