arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2511.02230cs.OScs.AIcs.NI
收藏

Continuum: 基于KV缓存生存时间的高效鲁棒多轮LLM智能体调度

Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live

UC Berkeley(加州大学伯克利分校);Stanford University(斯坦福大学);Tensormesh(Tensormesh公司);Tsinghua University(清华大学)

Hanchen Li, Runyuan He, Qiuyang Mang, Qizheng Zhang, Huanzhi Mao, Xiaokun Chen, Hangrui Zhou, Huanchen Zhang, Alvin Cheung, Joseph Gonzalez, Ion Stoica

AI总结:

针对多轮智能体工作负载中工具调用导致KV缓存无法跨轮重用的问题,提出Continuum系统,通过引入KV缓存的生存时间(TTL)机制,在GPU内存中选择性固定缓存,权衡重算/重载成本与排队延迟,实现作业完成时间平均提升8倍以上。

摘要:

KV缓存管理对于高效的LLM推理至关重要。为了最大化利用率,现有推理引擎会在新请求等待时驱逐已完成请求的KV缓存。但这种策略对于智能体工作负载不适用,因为智能体工作负载将LLM调用与工具调用交错进行,引入了停顿,从而阻止了跨轮次的KV有效重用。由于许多工具调用的持续时间远短于人类响应的多轮聊天,因此在工具调用期间保留KV缓存是有前景的。然而,仍存在许多挑战。首先,我们需要考虑重算或重载(如果启用卸载)的潜在成本,以及从GPU驱逐后增加的排队延迟。其次,由于工具调用持续时间的内部方差,该方法需要在工具调用持续时间有限可预测性下保持鲁棒性。我们提出了Continuum,一个通过引入KV缓存保留的生存时间(TTL)机制来优化多轮智能体工作负载作业完成时间的服务系统。对于生成工具调用的请求,Continuum选择性地将KV缓存固定在GPU内存中,其TTL值由重载成本和驱逐引起的潜在排队延迟决定。当TTL过期时,KV缓存可自动被驱逐以释放GPU内存,从而在边缘情况下提供鲁棒性能。当与程序级先来先服务结合时,Continuum保持了多轮连续性,并减少了智能体工作流的延迟。在真实世界智能体(SWE-Bench、BFCL、OpenHand)上使用Llama-3.1 8B/70B、Gemma-3 12B和GLM-4.5 355B的评估表明,Continuum在提高吞吐量的同时,将平均作业完成时间提升了8倍以上。

英文摘要:

KV cache management is essential for efficient LLM inference. To maximize utilization, existing inference engines evict finished requests' KV cache if new requests are waiting. This policy breaks for agentic workloads, which interleave LLM calls with tools, introducing pauses that prevent effective KV reuse across turns. Since many tool calls have much shorter durations than human response multi-turn chatbot, it would be promising to retain the KV cache in during these tools. However, many challenges remain. First, we need to consider both the potential cost of recomputation or reloading (if offloading enabled) as well as the increasing queueing delays after eviction from GPU. Second, due to the internal variance of tool call durations, the method needs to remain robust under limited predictability of tool call durations. We present Continnum, a serving system to optimize job completion time for multi-turn agent workloads by introducing time-to-live mechanism for KV cache retention. For requests that generate tool calls, Continnum selectively pins the KV cache in GPU memory with a time-to-live value determined by the reload cost and potential queueing delay induced by eviction. When the TTL expires, the KV cache can be automatically evicted to free up GPU memory, providing robust performance under edge cases. When combined with program-level first-come-first-serve, Continnum preserves multi-turn continuity, and reduces delay for agentic workflows. Evaluations on real-world agents (SWE-Bench, BFCL, OpenHand) with Llama-3.1 8B/70B, Gemma-3 12B, and GLM-4.5 355B shows that Continnum improves the average job completion times by over 8x while improving throughput.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑