ELUTQ: 在基于LUT的计算下优化量化精度以适应边缘大语言模型
ELUTQ: Optimizing Quantization Accuracy under LUT-Based Computation for Edge LLMs
AI总结:
ELUTQ通过分层线性量化提升边缘大语言模型低比特量化精度与性能
摘要:
权重量化有效降低了内存消耗并使大型语言模型能够部署在边缘设备上,但现有硬件友好的方法往往依赖于均匀量化,这在低比特设置下会导致权重分布拟合差和去量化开销高。在本文中,我们提出了ELUTQ,一种高效的量化框架,其特征是新颖的量化格式称为分层线性量化(HLQ)。HLQ旨在更好地捕捉权重的统计特性,并使用位串行LUT基于GEMM操作消除去量化开销。HLQ在低比特设置下显著提高了模型精度,并且在不重新训练权重的情况下实现了与QAT方法相当的性能。此外,ELUTQ集成了优化的量化流水线,使其能够仅使用64GB的CPU内存和48GB的VRAM完成LLaMA 3.1-70B的量化。为了使边缘设备上的高效部署,ELUTQ设计了高性能内核以支持端到端推理。我们的2位LLaMA3.1-8B在RTX 3090上实现了1.5倍于AWQ的速度提升。代码可在https://github.com/Nkniexin/ELUTQ上获得。
英文摘要:
Weight quantization effectively reduces memory consumption and enable the deployment of Large Language Models on edge devices, yet existing hardware-friendly methods often rely on uniform quantization, which suffers from poor weight-distribution fitting and high dequantization overhead under low-bit settings. In this paper, we propose ELUTQ, an efficient quantization framework featuring a novel quantization format termed Hierarchical Linear Quantization (HLQ). HLQ is designed to better capture the statistical characteristics of weights and eliminate dequantization overhead using Bit-serial LUT-based GEMM operations. HLQ significantly improves model accuracy under low-bit settings and achieves performance comparable to QAT methods without any retraining of the weights. Moreover, an optimized quantization pipeline is integrated into ELUTQ, enabling it to complete the quantization of LLaMA 3.1-70B using only 64 GB of CPU memory and 48 GB of VRAM, reducing the hardware requirements for large-scale model quantization. To enable efficient deployment on edge devices, ELUTQ designs high-performance kernels to support end-to-end inference. Our 2-bit LLaMA3.1-8B achieves 1.5x speedup over AWQ on RTX 3090. Code is available at https://github.com/Nkniexin/ELUTQ.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。