Scissorhands：利用重要性持久性假设实现测试时大语言模型KV缓存压缩
Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time
AI总结：
针对大语言模型KV缓存内存占用过高限制推理吞吐量的问题，提出基于重要性持久性假设的Scissorhands系统，无需微调即可将KV缓存压缩5倍，结合4位量化可达20倍压缩比且不损失模型质量。
摘要：
大型语言模型（LLMs）引发了新一轮令人兴奋的AI应用浪潮。大规模托管这些模型需要大量的内存资源。部署中的一个关键内存瓶颈源于上下文窗口。人们普遍认为模型权重是内存密集型的；然而，在生成过程中存储的键值嵌入（KV缓存）的大小很容易超过模型大小。KV缓存的巨大规模限制了推理批处理大小，而这对于高吞吐量推理工作负载至关重要。受注意力分数一个有趣观察的启发，我们假设重要性持久性：只有在一开始具有重大影响的关键标记，才会对未来生成产生显著影响。基于对这一假设的经验验证和理论分析，我们提出了Scissorhands，一个无需微调模型即可将KV缓存内存使用维持在固定预算内的系统。本质上，Scissorhands通过以较高概率存储关键标记来管理KV缓存。我们验证了Scissorhands在不损害模型质量的情况下，将KV缓存的推理内存使用减少了高达5倍。我们进一步证明，Scissorhands可以与传统上用于压缩模型权重的4位量化相结合，实现高达20倍的压缩。
英文摘要：
Large language models(LLMs) have sparked a new wave of exciting AI applications. Hosting these models at scale requires significant memory resources. One crucial memory bottleneck for the deployment stems from the context window. It is commonly recognized that model weights are memory hungry; however, the size of key-value embedding stored during the generation process (KV cache) can easily surpass the model size. The enormous size of the KV cache puts constraints on the inference batch size, which is crucial for high throughput inference workload. Inspired by an interesting observation of the attention scores, we hypothesize the persistence of importance: only pivotal tokens, which had a substantial influence at one step, will significantly influence future generations. Based on our empirical verification and theoretical analysis around this hypothesis, we propose Scissorhands, a system that maintains the memory usage of the KV cache at a fixed budget without finetuning the model. In essence, Scissorhands manages the KV cache by storing the pivotal tokens with a higher probability. We validate that Scissorhands reduces the inference memory usage of the KV cache by up to 5X without compromising model quality. We further demonstrate that Scissorhands can be combined with 4-bit quantization, traditionally used to compress model weights, to achieve up to 20X compression.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
