残差流足矣:关于Transformer推理中KV缓存冗余性的问题
The Residual Stream Is All You Need: On the Redundancy of the KV Cache in Transformer Inference
AI总结:
本文证明Transformer推理中的KV缓存是冗余的,残差流可完全替代其功能,通过重新计算残差流可实现零误差,且提出KV-Direct方法在低内存下实现高效推理。
摘要:
关键值(KV)缓存通常被视为Transformer推理中的必要状态,大量工作致力于压缩、驱逐或近似其条目。我们证明该状态是完全冗余的:每层的键和值都是残差流的确定性投影,从单个残差向量重新计算可实现零重建误差,而非近似。我们验证了六个模型(135M至4B参数)的这一结论。每层的跨任务残差修补使修补与原始输出分布的D_KL为零,证实残差流满足马尔可夫性质且是唯一信息承载状态。完全移除缓存并从头重新计算,在所有测试模型上通过贪婪解码获得完全一致的输出。我们基于此结果提出KV-Direct,一种低内存推理方案,通过检查残差向量(Gemma 3-4B模型每token 5KB)而非完整KV对(136KB)来实现。在20轮对话中,KV-Direct在峰值内存42MB时保持100%的token匹配,而标准缓存超过103MB。与五个驱逐基线(H2O、StreamingLLM、SnapKV、TOVA、仅窗口)相比,KV-Direct在所有缓存预算下保持100%的token匹配,而所有基线下降至5-28%。每操作延迟分析显示,在中等批量大小下,重新计算比读取缓存张量快5倍。代码可在https://github.com/Kaleemullahqasim/KV-Direct获取。
英文摘要:
The key-value (KV) cache is widely treated as essential state in transformer inference, and a large body of work engineers policies to compress, evict, or approximate its entries. We prove that this state is entirely redundant: keys and values at every layer are deterministic projections of the residual stream, and recomputing them from a single residual vector per token incurs exactly zero reconstruction error, not approximately, but bit-identically. We verify this across six models from four architecture families (135M to 4B parameters). Cross-task residual patching at every layer produces D_KL = 0 between patched and original output distributions, confirming that the residual stream satisfies a Markov property and is the sole information-carrying state. Removing the cache entirely and recomputing from scratch yields token-identical output under greedy decoding on all models tested. We build on this result with KV-Direct, a bounded-memory inference scheme that checkpoints residual vectors (5 KB per token on Gemma 3-4B) instead of full KV pairs (136 KB), recomputing keys and values on demand. Over 20 conversation turns, KV-Direct holds peak memory at 42 MB while the standard cache grows past 103 MB. Against five eviction baselines (H2O, StreamingLLM, SnapKV, TOVA, window-only), KV-Direct maintains 100% token match at every cache budget; all baselines degrade to 5-28%. A per-operation latency analysis shows recomputation runs up to 5x faster than reading cached tensors at moderate batch sizes. Code is available at https://github.com/Kaleemullahqasim/KV-Direct.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。