KV cache management is essential for efficient LLM inference. To maximize utilization, existing inference engines evict finished requests' KV cache if new requests are waiting. This policy breaks for agentic workloads, which interleave LLM calls with tools, introducing pauses that prevent effective KV reuse across turns. Since many tool calls have much shorter durations than human response multi-turn chatbot, it would be promising to retain the KV cache in during these tools. However, many challenges remain. First, we need to consider both the potential cost of recomputation or reloading (if offloading enabled) as well as the increasing queueing delays after eviction from GPU. Second, due to the internal variance of tool call durations, the method needs to remain robust under limited predictability of tool call durations.
We present Continnum, a serving system to optimize job completion time for multi-turn agent workloads by introducing time-to-live mechanism for KV cache retention. For requests that generate tool calls, Continnum selectively pins the KV cache in GPU memory with a time-to-live value determined by the reload cost and potential queueing delay induced by eviction. When the TTL expires, the KV cache can be automatically evicted to free up GPU memory, providing robust performance under edge cases. When combined with program-level first-come-first-serve, Continnum preserves multi-turn continuity, and reduces delay for agentic workflows. Evaluations on real-world agents (SWE-Bench, BFCL, OpenHand) with Llama-3.1 8B/70B, Gemma-3 12B, and GLM-4.5 355B shows that Continnum improves the average job completion times by over 8x while improving throughput.
Online Scheduling for LLM Inference with KV Cache Constraints
Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology(麻省理工学院电子工程与计算机科学系);HKUST(香港科技大学);Microsoft Research(微软研究院);Sloan School of Management, Massachusetts Institute of Technology(斯隆管理学院,麻省理工学院);Operations Research Center, Massachusetts Institute of Technology(运营研究中心,麻省理工学院)
Optimizing LLM Inference: Fluid-Guided Online Scheduling with Memory Constraints
Institute for Data, Systems, and Society, Massachusetts Institute of Technology(数据、系统与社会研究所,麻省理工学院);School of Mathematical Sciences, Peking University(北京大学数学科学学院);Alibaba Group(阿里巴巴集团)
Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization
School of Computing and Information Systems, The University of Melbourne(墨尔本大学计算与信息系统学院);School of Computer Science and Technology, Huazhong University of Science and Technology(华中科技大学计算机科学与技术学院)
A Survey on Large Language Model Acceleration based on KV Cache Management
Department of Computing, The Hong Kong Polytechnic University(计算系,香港理工大学);Department of Computer Science and Engineering(计算机科学与工程系);Department of Computer Science and Technology(计算机科学与技术系);Department of Computing and Data Science(计算与数据科学系)
GroupKV: Hierarchical KV Cache Management for Long-Context Diffusion LLM Inference
Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所);University of Chinese Academy of Sciences(中国科学院大学);Nanjing University of Science and Technology(南京理工大学)