Large language model agents execute programs comprising multiple model turns interleaved with external tool calls. Their job completion time depends on how the serving system retains KV state across tool gaps, routes successor turns, and schedules competing programs. Most existing serving simulators operate on request streams in which arrivals are externally supplied and KV state follows request- or cache-scoped semantics. They therefore cannot jointly represent the cross-turn state and policy-dependent successor releases needed to evaluate counterfactual agent-serving trajectories. We present AgentServeSim, a simulator whose unit of execution is the agent program. A Program Control Block maintains cross-turn state, while a Program Orchestrator causally releases successor turns from simulated predecessor completions. A Retention Plane controls KV state across tool gaps, and a Dispatch Plane determines where and when each ready turn executes. We validate AgentServeSim against real vLLM deployments in 20 paired simulator-real cells spanning two GPU platforms, Llama-3.1-8B and Llama-3.1-70B, coding and function-calling agents, and five arrival rates. Mean JCT error remains within 5.5% on B200 and 5.2% in the saturated RTX PRO 6000 regime. Finally, we propose LLM-driven automated agent-serving policy search using AgentServeSim as a CPU-based fitness evaluator. The resulting policies improve mean JCT over hand-written seed policies by 0.5% for KV retention and 2.8% for scheduling.
Recency/Frequency Adaptive KV Caching for Large Language Model Serving
Department of Computer Science, Johns Hopkins University, Baltimore, USA(约翰霍普金斯大学计算机科学系);Argonne National Laboratory, Lemont, USA(阿贡国家实验室);Parasail, Inc., San Mateo, USA(Parasail公司)
Reasoning Compression with Mixed-Policy Distillation
Technical University of Munich(慕尼黑技术大学);GESIS – Leibniz Institute for the Social Sciences(莱布尼茨社会科学研究所);Northeastern University(东北大学);LMU Munich(慕尼黑大学);Siemens AG(西门子股份公司);University of Cambridge(剑桥大学);University of Oxford(牛津大学);Mina AI
Trainable Smooth-Rotation Transforms with Learned Channel Scales for LLM Quantization
Doctoral School of Applied Informatics and Applied Mathematics, Obuda University(应用信息学与应用数学博士学校,奥布达大学);John von Neumann Faculty of Informatics, Obuda University(约翰·冯·诺伊曼信息学系,奥布达大学)