VecInfer:通过异常值抑制向量量化实现低比特KV缓存的高效LLM推理
VecInfer: Efficient LLM Inference with Low-Bit KV Cache via Outlier-Suppressed Vector Quantization
AI总结:
VecInfer通过平滑和Hadamard变换抑制键缓存异常值,实现低比特KV缓存压缩,在2比特量化下达到全精度性能,并显著加速推理。
摘要:
键值(KV)缓存在大语言模型(LLM)推理过程中引入了大量的内存开销。尽管现有的向量量化(VQ)方法减少了KV缓存的使用,并在不同位宽下提供了灵活的表示能力,但在超低位宽下,由于键缓存中的异常值阻碍了码本的有效利用,这些方法会遭受严重的性能下降。为了解决这一挑战,我们提出了VecInfer,一种新颖的VQ方法,用于激进的KV缓存压缩,同时实现高效推理。通过应用平滑变换和Hadamard变换,VecInfer抑制了键缓存中的异常值,使码本能够全面覆盖原始数据分布,从而降低量化难度。为了便于高效部署,我们设计了一个优化的CUDA内核,将计算与反量化融合,以最小化内存访问开销。大量评估表明,VecInfer在长上下文理解和数学推理任务中均持续优于现有的量化基线。仅使用2比特量化,VecInfer就能达到与全精度相当的性能,同时在Llama-3.1-8B上,对于196k序列长度,在大批量自注意力计算中实现了高达2.7倍的加速,在单批次端到端延迟中实现了8.3倍的降低。
英文摘要:
The Key-Value (KV) cache introduces substantial memory overhead during large language model (LLM) inference. Although existing vector quantization (VQ) methods reduce KV cache usage and provide flexible representational capacity across bit-widths, they suffer severe performance degradation at ultra-low bit-widths due to key cache outliers that hinder effective codebook utilization. To address this challenge, we propose VecInfer, a novel VQ method for aggressive KV cache compression while enabling efficient inference. By applying smooth and Hadamard transformations, VecInfer suppresses outliers in the key cache, enabling the codebook to comprehensively cover the original data distribution and thereby reducing quantization difficulty. To facilitate efficient deployment, we design an optimized CUDA kernel that fuses computation with dequantization to minimize memory access overhead. Extensive evaluations demonstrate that VecInfer consistently outperforms existing quantization baselines across both long-context understanding and mathematical reasoning tasks. With only 2-bit quantization, VecInfer achieves performance comparable to full precision, while delivering up to $\mathbf{2.7\times}$ speedup in large-batch self-attention computation and $\mathbf{8.3\times}$ reduction in single-batch end-to-end latency on Llama-3.1-8B with a 196k sequence length.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。