DynamicKV：面向长上下文LLM的任务感知自适应KV缓存压缩
DynamicKV: Task-Aware Adaptive KV Cache Compression for Long Context LLMs
AI总结：
DynamicKV通过任务感知的自适应层间KV缓存分配，在仅保留1.7%缓存时达到85%性能，并在极端压缩下超越SOTA 11%。
摘要：
在LLM中，高效的KV缓存管理对于RAG和摘要等长上下文任务至关重要。现有的KV缓存压缩方法强制执行固定模式，忽略了任务特定特征，减少了对关键信息的保留。然而，我们观察到不同任务中各层的激活模式存在明显差异，这凸显了针对每个任务独特需求定制自适应策略的必要性。基于这一洞察，我们提出了DynamicKV，一种通过调整每层保留的token数量来动态优化token保留以适应特定任务的方法。DynamicKV建立了全局和每层的最大KV缓存预算，临时保留当前层的最大预算，并在推理过程中定期更新所有先前层的KV缓存大小。我们的方法仅保留1.7%的KV缓存大小，同时在LongBench上实现了约85%的完整KV缓存性能。值得注意的是，即使在极端压缩（0.9%）下，DynamicKV在Mistral-7B-Instruct-v0.2的Needle-in-a-Haystack测试中仍比最先进（SOTA）方法高出11%。代码将发布。
英文摘要：
Efficient KV cache management in LLMs is crucial for long-context tasks like RAG and summarization. Existing KV cache compression methods enforce a fixed pattern, neglecting task-specific characteristics and reducing the retention of essential information. However, we observe distinct activation patterns across layers in various tasks, highlighting the need for adaptive strategies tailored to each task's unique demands. Based on this insight, we propose DynamicKV, a method that dynamically optimizes token retention by adjusting the number of tokens retained at each layer to adapt to the specific task. DynamicKV establishes global and per-layer maximum KV cache budgets, temporarily retaining the maximum budget for the current layer, and periodically updating the KV cache sizes of all preceding layers during inference. Our method retains only 1.7% of the KV cache size while achieving ~85% of the Full KV cache performance on LongBench. Notably, even under extreme compression (0.9%), DynamicKV surpasses state-of-the-art (SOTA) methods by 11% in the Needle-in-a-Haystack test using Mistral-7B-Instruct-v0.2. The code will be released.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
