TaDA：一种免训练的自适应KV缓存压缩与均值中心化解码方法
TaDA: Training-free recipe for Decoding with Adaptive KV Cache Compression and Mean-centering
AI总结：
TaDA提出一种免训练的KV缓存量化压缩方法，通过按层自适应调整量化精度并结合均值中心化消除离群值单独处理，将内存占用降至16位基线的27%且精度可比。
摘要：
Transformer模型中的键值（KV）缓存是实现高效解码或推理的关键组件，然而其内存需求随序列长度增长而急剧上升，给大语言模型的可扩展部署带来了重大挑战。在多种KV缓存压缩方法中，键和值激活的量化已被广泛探索。大多数KV缓存量化方法仍需要单独处理稀疏且非连续的离群值。为解决这一问题，我们提出了TaDA，一种免训练的KV缓存压缩方法，其量化精度能够根据各层的误差敏感度自适应调整，并通过均值中心化消除对离群值的单独处理。我们的方法在多个支持不同上下文长度的模型上取得了显著的精度提升。此外，我们的方法无需单独管理离群元素——这是大多数传统量化方法中长期存在的障碍。在标准基准上的实验表明，我们的技术将KV缓存内存占用降低至原始16位基线的27%，同时实现了可比的精度。我们的方法通过潜在地支持更长上下文长度模型、推理模型以及更长思维链的推理，为语言模型中可扩展且高性能的推理铺平了道路。
英文摘要：
The key-value (KV) cache in transformer models is a critical component for efficient decoding or inference, yet its memory demands scale poorly with sequence length, posing a major challenge for scalable deployment of large language models. Among several approaches to KV cache compression, quantization of key and value activations has been widely explored. Most KV cache quantization methods still need to manage sparse and noncontiguous outliers separately. To address this, we introduce TaDA, a training-free recipe for KV cache compression with quantization precision that adapts to error sensitivity across layers and a mean centering to eliminate separate outlier handling. Our approach yields substantial accuracy improvements for multiple models supporting various context lengths. Moreover, our approach does not need to separately manage outlier elements -- a persistent hurdle in most traditional quantization methods. Experiments on standard benchmarks demonstrate that our technique reduces KV cache memory footprint to 27% of the original 16-bit baseline while achieving comparable accuracy. Our method paves the way for scalable and high-performance reasoning in language models by potentially enabling inference for longer context length models, reasoning models, and longer chain of thoughts.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
