arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.10238cs.CLcs.LG
收藏

学习从键值缓存中驱逐

Learning to Evict from Key-Value Cache

Luca Moschella, Laura Manduchi, Ozan Sener

AI总结:

提出基于强化学习的KV缓存驱逐策略KVP,通过轻量级逐头代理学习预测令牌未来效用,显著提升长上下文和多轮对话性能。

摘要:

大型语言模型(LLM)的规模不断增长,使得高效推理面临挑战,这主要是由于自回归键值(KV)缓存的内存需求。现有的驱逐或压缩方法虽然降低了成本,但依赖于启发式规则,如最近使用或过去的注意力分数,这些仅作为令牌未来效用的间接代理,并引入了计算开销。我们将KV缓存驱逐重新定义为强化学习(RL)问题:学习根据令牌对未来解码的预测有用性进行排序。为此,我们引入了KV策略(KVP),这是一个轻量级的逐头RL代理框架,使用仅包含键和值向量的预计算生成轨迹进行训练。每个代理学习一个专门的驱逐策略,由来自未来效用的整体奖励指导,该奖励评估所有缓存预算下的排序质量,无需修改底层LLM或额外推理。在长上下文基准RULER(高达128K令牌)和多轮对话基准OASST2-4k上对两个模型系列进行评估,KVP显著优于强基线。在标准下游任务(BoolQ、LongBench段落检索、GovReport)上的零样本测试进一步表明,KVP能够泛化到其训练分布之外以及更长的序列长度。这些结果表明,学习预测未来令牌效用是自适应KV缓存管理的一种强大且可扩展的范式。

英文摘要:

The growing size of Large Language Models (LLMs) makes efficient inference challenging, primarily due to the memory demands of the autoregressive Key-Value (KV) cache. Existing eviction or compression methods reduce cost but rely on heuristics, such as recency or past attention scores, which serve only as indirect proxies for a token's future utility and introduce computational overhead. We reframe KV cache eviction as a reinforcement learning (RL) problem: learning to rank tokens by their predicted usefulness for future decoding. To this end, we introduce KV Policy (KVP), a framework of lightweight per-head RL agents trained on pre-computed generation traces using only key and value vectors. Each agent learns a specialized eviction policy guided by a holistic reward, derived from future utility, that evaluates the quality of the ranking across all cache budgets, requiring no modifications to the underlying LLM or additional inference. Evaluated across two model families on the long-context benchmark RULER (up to 128K tokens) and the multi-turn dialogue benchmark OASST2-4k, KVP significantly outperforms strong baselines. Zero-shot tests on standard downstream tasks (BoolQ, LongBench passage retrieval, GovReport) further show that KVP generalizes beyond its training distribution and to considerably longer sequence lengths. These results demonstrate that learning to predict future token utility is a powerful and scalable paradigm for adaptive KV cache management.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑