SVDq：面向大语言模型注意力的1.25比特且410倍的键缓存压缩
SVDq: 1.25-bit and 410x Key Cache Compression for LLM Attention
AI总结：
针对大语言模型KV缓存高效压缩需求，提出基于SVD的K缓存混合精度量化方法SVDq，可实现1.25比特等效精度，结合键稀疏化最高达410倍压缩比且性能几乎无损。
摘要：
为实现大语言模型（LLM）的高效推理，键值（KV）缓存的有效压缩至关重要。目前已明确三类主要的KV缓存压缩技术，即稀疏化、通道压缩和量化。本研究提出SVDq，一种基于奇异值分解（SVD）的K缓存混合精度量化方法。首先，利用SVD基表示将K缓存转换为潜在通道。由于潜在通道中的值衰减极快，仅在少数几个潜在通道后就可忽略不计，因此本方法随后对潜在通道引入重要性感知量化与压缩，从而能为更重要的通道有效分配更高精度。理论上，我们证明SVDq产生的量化误差（为原空间逐通道键量化的0.1倍甚至更低）远低于原空间中的逐通道键量化误差。我们基于RULER和LongBench基准的研究结果表明，SVDq可实现低至1.25比特的等效键缓存精度。当与键稀疏化结合时，它在注意力计算中可达到最高410倍的键压缩比，同时保持相当的模型性能。值得注意的是，我们的方法在LongBench数据集上几乎无损。这表明SVDq实现了高精度低比特量化，为LLM中的KV缓存压缩提供了更高效的解决方案。
英文摘要：
For the efficient inference of Large Language Models (LLMs), the effective compression of key-value (KV) cache is essential. Three main types of KV cache compression techniques, namely sparsity, channel compression, and quantization, have been identified. This study presents SVDq, a Singular Value Decomposition (SVD) - based mixed precision quantization method for K cache. Initially, K cache is transformed into latent channels using SVD basis representations. Since the values in latent channels decay rapidly and become negligible after only a few latent channels, our method then incorporates importance-aware quantization and compression for latent channels. This enables the effective allocation of higher precision to more significant channels. Theoretically, we prove that SVDq results in quantization errors (x0.1 or even lower) that are much lower than those of per-channel key quantization in the original space. Our findings based on RULER and LongBench benchmarks demonstrate that SVDq can achieve an equivalent key cache precision as low as 1.25-bit. When combined with key sparsity, it can reach a key compression ratio of up to 410x for attention computation, all while maintaining comparable model performance. Notably, our method is nearly lossless for LongBench datasets. This indicates that SVDq enables high-precision low-bit quantization, providing a more efficient solution for KV cache compression in LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
