RILQ：基于秩不敏感LoRA的量化误差补偿方法，用于提升2比特大语言模型精度
RILQ: Rank-Insensitive LoRA-based Quantization Error Compensation for Boosting 2-bit Large Language Model Accuracy
AI总结：
提出RILQ方法，利用模型级激活差异损失的秩不敏感特性，跨层协同调整LoRA适配器，以补偿2比特量化误差，在LLaMA-2/3上显著提升精度并保持计算效率。
摘要：
低秩适配（LoRA）已成为参数高效大语言模型微调的主流方法，基于LoRA的量化误差补偿（LQEC）作为恢复压缩大语言模型精度的有力工具应运而生。然而，LQEC在低于4比特的场景下表现不佳，且此前尚无研究探究其局限性。我们提出RILQ（秩不敏感的基于LoRA的量化误差补偿）以理解其根本限制并提升2比特大语言模型精度。基于揭示模型级激活差异损失秩不敏感特性的秩分析，RILQ采用该损失跨层协同调整适配器，从而以低秩适配器实现稳健的误差补偿。在LLaMA-2和LLaMA-3上的评估表明，RILQ在各种最先进量化器下均能持续提升2比特量化推理性能，并在任务特定微调中增强精度。RILQ保持了与现有LoRA方法相当的计算效率，支持适配器合并的权重量化大语言模型推理，同时显著提升精度，使其成为提升2比特大语言模型性能的有前景方法。我们的代码可在https://github.com/aiha-lab/RILQ获取。
英文摘要：
Low-rank adaptation (LoRA) has become the dominant method for parameter-efficient LLM fine-tuning, with LoRA-based quantization error compensation (LQEC) emerging as a powerful tool for recovering accuracy in compressed LLMs. However, LQEC has underperformed in sub-4-bit scenarios, with no prior investigation into understanding this limitation. We propose RILQ (Rank-Insensitive LoRA-based Quantization Error Compensation) to understand fundamental limitation and boost 2-bit LLM accuracy. Based on rank analysis revealing model-wise activation discrepancy loss's rank-insensitive nature, RILQ employs this loss to adjust adapters cooperatively across layers, enabling robust error compensation with low-rank adapters. Evaluations on LLaMA-2 and LLaMA-3 demonstrate RILQ's consistent improvements in 2-bit quantized inference across various state-of-the-art quantizers and enhanced accuracy in task-specific fine-tuning. RILQ maintains computational efficiency comparable to existing LoRA methods, enabling adapter-merged weight-quantized LLM inference with significantly enhanced accuracy, making it a promising approach for boosting 2-bit LLM performance. Our code is available at https://github.com/aiha-lab/RILQ.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
