SCOPE：长上下文生成中键值缓存压缩的优化
SCOPE: Optimizing Key-Value Cache Compression in Long-context Generation
AI总结：
SCOPE框架针对长上下文生成中KV缓存瓶颈，提出预填充与解码阶段分离优化策略，通过滑动选择关键重击者并采用自适应不连续策略优化内存，在LongGenBench上验证了其有效性与插件兼容性。
摘要：
键值（KV）缓存已成为大语言模型（LLM）在长上下文生成中的瓶颈。尽管该领域已有大量努力，但解码阶段的优化通常被忽视。然而，我们认为这种优化至关重要，尤其是对于长输出生成任务，基于以下两个观察：（i）在预填充阶段进行过度压缩需要特定的完整上下文，这会损害推理任务的理解能力；（ii）在长输出的推理任务中会出现重击者（heavy hitters）的偏移。因此，我们引入了SCOPE，一个简单而高效的框架，在预填充阶段和解码阶段分别执行KV缓存优化。具体而言，预填充阶段的KV缓存被保留以维持关键信息，同时提出了一种基于滑动的新策略来为解码阶段选择关键的重击者。通过自适应和不连续策略进一步优化了内存使用和内存传输。在LongGenBench上的大量实验表明，SCOPE的有效性和泛化能力，以及它作为插件与其他仅预填充KV压缩方法的兼容性。
英文摘要：
Key-Value (KV) cache has become a bottleneck of LLMs for long-context generation. Despite the numerous efforts in this area, the optimization for the decoding phase is generally ignored. However, we believe such optimization is crucial, especially for long-output generation tasks based on the following two observations: (i) Excessive compression during the prefill phase, which requires specific full context impairs the comprehension of the reasoning task; (ii) Deviation of heavy hitters occurs in the reasoning tasks with long outputs. Therefore, SCOPE, a simple yet efficient framework that separately performs KV cache optimization during the prefill and decoding phases, is introduced. Specifically, the KV cache during the prefill phase is preserved to maintain the essential information, while a novel strategy based on sliding is proposed to select essential heavy hitters for the decoding phase. Memory usage and memory transfer are further optimized using adaptive and discontinuous strategies. Extensive experiments on LongGenBench show the effectiveness and generalization of SCOPE and its compatibility as a plug-in to other prefill-only KV compression methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
