发表机构
UC Santa Barbara; LinkedIn（加州大学圣巴巴拉分校; 领英公司）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
该研究针对LLM智能体的KV缓存瓶颈，实证对比了令牌驱逐与注意力匹配两类在线压缩方法，发现延迟压缩可恢复性能，令牌驱逐在代理不完善时更鲁棒，能大幅压缩KV缓存并提升吞吐量。
AI 中文摘要
LLM智能体积累了推理步骤、工具调用和环境反馈的长轨迹，使得KV缓存成为主要推理瓶颈。KV缓存压缩可降低该成本，但大多数现有方法假设上下文是静态的，即未来查询已知或可离线近似。而智能体需要在线压缩：必须在未来相关性未知前压缩新信息，使用推理路径上足够廉价的代理查询。我们研究了令牌驱逐（TE）和注意力匹配（AM）两类在线压缩方法，将二者适配于压缩智能体轮次，并对比了边界、重复预填充、延迟未来生成查询等廉价代理源。在BrowseComp-Plus和WideSearch上的实验表明，即时压缩往往会损害性能，而延迟压缩以使用智能体的未来查询可恢复大部分性能差距；此外，在代理不完善的情况下，TE通常比AM更鲁棒。在不同规模的模型上，TE可在保留大部分准确率的同时将KV缓存减少80%，且相比无压缩基线可提高吞吐量。这些结果表明，代理查询选择是实用在线KV压缩的核心设计选择。
英文摘要
LLM agents accumulate long trajectories of reasoning steps, tool calls, and environment feedback, making the KV cache a major inference bottleneck. KV cache compaction can reduce this cost, but most prior methods assume a static context where future queries are known or can be approximated offline. Agents instead require online compaction: new information must be compressed before future relevance is known, using proxy queries cheap enough for the inference path. We study online compaction across token eviction (TE) and attention matching (AM), adapting both to compact agent turns and comparing cheap proxy sources such as boundary, repeat-prefill, and delayed future-generation queries. Experiments on BrowseComp-Plus and WideSearch show that immediate compaction often hurts performance, whereas delaying compaction to use the agent's future queries recovers much of the gap. Moreover, TE is often more robust than AM under imperfect proxies. Across models at different scales, TE preserves most of the accuracy while reducing KV cache by 80%, and can improve throughput over the no compaction baseline. These results position proxy-query selection as a core design choice for practical online KV compaction.
