Kara: 通过滑动窗口KV缓存压缩实现高效推理LLM服务
KARA: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compression
AI总结：
针对推理语言模型长思维链导致KV缓存过大、解码延迟高的问题，提出Kara滑动窗口KV缓存压缩方法，利用双向注意力评分选择信息性KV对并通过Token2Chunk模块扩展为块，结合PagedAttention构建KvLLM推理框架，降低内存占用并提升输出吞吐量。
摘要：
推理语言模型通常生成长思维链（CoT），在解码阶段积累大量KV缓存，导致高解码延迟和有限的吞吐量。为了解决这些问题，KV缓存压缩已成为一种有前景的技术，通过选择性移除不重要的KV对同时保留有用的用于后续解码，从而减少内存开销。然而，我们识别出现有KV缓存压缩方法的两个关键限制：1）它们的阈值触发压缩策略可能提供有限的吞吐量提升甚至降低吞吐量，并且可能完全消除序列某些块中的KV对，从而可能加剧信息损失。2）它们通常保留孤立的KV对或具有刚性边界的固定大小块，无法在任意标记位置保留重要的灵活大小块。为了克服这些限制，我们提出Kara，一种滑动窗口KV缓存压缩方法，通过仅对最近生成的上下文进行操作来执行解码时压缩。Kara利用双向注意力来评分并选择窗口中的信息性KV对。为了灵活保留重要的语义信息，我们设计了一个Token2Chunk模块，将选中的KV对子集扩展为块。此外，我们将Kara适配到PagedAttention，并开发了KvLLM，一个基于vLLM构建的推理框架，它减少了KV缓存内存使用并有效提高了输出吞吐量。大量实验证明了所提出的Kara和KvLLM的一致性能改进。
英文摘要：
Reasoning language models often generate long chain-of-thought (CoT), which accumulates a massive KV cache during the decoding phase and incurs high decoding latency and limited throughput. To address these issues, KV cache compression has emerged as a promising technique for reducing memory overhead by selectively removing unimportant KV pairs while preserving useful ones for subsequent decoding. Nevertheless, we identify two key limitations in existing KV cache compression methods: 1) their threshold-triggered compression policy may provide limited throughput improvement or even reduce throughput, and may fully eliminate KV pairs from certain blocks of the sequence, potentially worsening information loss. 2) they typically retain either isolated KV pairs or fixed-size chunks with rigid boundaries, failing to preserve important flexible-sized chunks at arbitrary token positions. To overcome these limitations, we propose Kara, a sliding-window KV cache compression method that performs decoding-time compression by operating only on the recently generated context. Kara leverages bidirectional attention to score and select informative KV pairs in the window. To enable flexible preservation of important semantic information, we design a Token2Chunk module to expand a subset of selected KV pairs into chunks. Furthermore, we adapt Kara to PagedAttention and develop KvLLM, an inference framework built upon vLLM, which reduces KV cache memory usage and effectively improves output throughput. Extensive experiments demonstrate consistent performance improvements of proposed Kara and KvLLM.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
