DartQuant：面向LLM量化的高效旋转分布校准
DartQuant: Efficient Rotational Distribution Calibration for LLM Quantization
AI总结：
DartQuant提出分布感知旋转校准与QR-Orth优化方案，在70B模型上实现47倍加速和10倍显存节省，并首次在单张3090 GPU上完成旋转校准。
摘要：
量化在加速大规模模型推理中起着至关重要的作用，旋转矩阵已被证明能够通过平滑离群值有效提升量化性能。然而，旋转优化算法的端到端微调会带来高昂的计算成本，并且容易过拟合。为应对这一挑战，我们提出了一种高效的分布感知旋转校准方法DartQuant，该方法通过约束旋转后激活值的分布来降低旋转优化的复杂度。这一方法还有效减少了对任务特定损失的依赖，从而降低了过拟合风险。此外，我们引入了QR-Orth优化方案，用更高效的求解方式替代了昂贵的交替优化。在多种模型量化实验中，DartQuant展现出优越的性能。与现有方法相比，在70B模型上，它实现了旋转优化47$\ imes$的加速和10$\ imes$的显存节省。此外，它首次成功在单张3090 GPU上完成70B模型的旋转校准，使大语言模型的量化在资源受限环境中变得可行。代码可在https://github.com/CAS-CLab/DartQuant.git获取。
英文摘要：
Quantization plays a crucial role in accelerating the inference of large-scale models, and rotational matrices have been shown to effectively improve quantization performance by smoothing outliers. However, end-to-end fine-tuning of rotational optimization algorithms incurs high computational costs and is prone to overfitting. To address this challenge, we propose an efficient distribution-aware rotational calibration method, DartQuant, which reduces the complexity of rotational optimization by constraining the distribution of the activations after rotation. This approach also effectively reduces reliance on task-specific losses, thereby mitigating the risk of overfitting. Additionally, we introduce the QR-Orth optimization scheme, which replaces expensive alternating optimization with a more efficient solution. In a variety of model quantization experiments, DartQuant demonstrates superior performance. Compared to existing methods, it achieves 47$\times$ acceleration and 10$\times$ memory savings for rotational optimization on a 70B model. Furthermore, it is the first to successfully complete rotational calibration for a 70B model on a single 3090 GPU, making quantization of large language models feasible in resource-constrained environments. Code is available at https://github.com/CAS-CLab/DartQuant.git.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
