KVQuant:通过 KV 缓存量化迈向 1000 万上下文长度的 LLM 推理
KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization
AI总结:
KVQuant 提出多种低精度 KV 缓存量化新方法,在 3-bit 下实现极低困惑度下降,支持千万级上下文推理并带来显著加速。
摘要:
LLMs 在需要大上下文窗口的应用中越来越广泛使用,而在这些大上下文窗口中,KV 缓存激活成为推理期间内存消耗的主要来源。量化是压缩 KV 缓存激活的一种有前景的方法;然而,现有解决方案无法在低于 4-bit 精度下准确表示激活。我们的工作 KVQuant 通过引入几种新方法来促进低精度 KV 缓存量化:(i) 通道级 Key 量化,我们调整量化 Key 激活的维度以更好地匹配分布;(ii) Pre-RoPE Key 量化,我们在旋转位置嵌入之前量化 Key 激活以减轻其对量化的影响;(iii) 非均匀 KV 缓存量化,我们推导出逐层敏感度加权的非均匀数据类型以更好地表示分布;(iv) 向量级密集与稀疏量化,我们为每个向量单独隔离异常值以最小化量化范围的偏斜。通过将我们的方法应用于 LLaMA、Llama-2、Llama-3 和 Mistral 模型,我们在 Wikitext-2 和 C4 上使用 3-bit 量化实现了小于 0.1 的困惑度下降,优于现有方法。我们的方法能够在单个 A100-80GB GPU 上为 LLaMA-7B 提供长达 100 万的上下文长度,并在 8-GPU 系统上提供长达 1000 万的上下文长度。我们为 KVQuant 开发了自定义 CUDA 内核,对于 LLaMA-7B 模型,与基线 fp16 矩阵-向量乘法相比,我们能够实现高达约 1.7 倍的加速。
英文摘要:
LLMs are seeing growing use for applications which require large context windows, and with these large context windows KV cache activations surface as the dominant contributor to memory consumption during inference. Quantization is a promising approach for compressing KV cache activations; however, existing solutions fail to represent activations accurately in sub-4-bit precision. Our work, KVQuant, facilitates low precision KV cache quantization by incorporating several novel methods: (i) Per-Channel Key Quantization, where we adjust the dimension along which we quantize the Key activations to better match the distribution; (ii) Pre-RoPE Key Quantization, where we quantize Key activations before the rotary positional embedding to mitigate its impact on quantization; (iii) Non-Uniform KV Cache Quantization, where we derive per-layer sensitivity-weighted non-uniform datatypes that better represent the distributions; and (iv) Per-Vector Dense-and-Sparse Quantization, where we isolate outliers separately for each vector to minimize skews in quantization ranges. By applying our method to the LLaMA, Llama-2, Llama-3, and Mistral models, we achieve < 0.1 perplexity degradation with 3-bit quantization on both Wikitext-2 and C4, outperforming existing approaches. Our method enables serving LLaMA-7B with a context length of up to 1 million on a single A100-80GB GPU and up to 10 million on an 8-GPU system. We develop custom CUDA kernels for KVQuant, showing that we can achieve up to ~1.7x speedups, compared to baseline fp16 matrix-vector multiplications, for the LLaMA-7B model.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。