arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.09613cs.CLcs.AI
收藏

AGENTSERVESIM:面向多轮LLM智能体服务的硬件感知模拟器

AgentServeSim: Serving-System Simulation and Policy Search for LLM Agent Programs

University of Central Florida(中佛罗里达大学)

Rakibul Hasan Rajib, Mengxin Zheng, Qian Lou

AI总结:

提出AGENTSERVESIM模拟器,通过程序编排器、工具模拟器、会话感知路由器和KV驻留模型等模块,在程序粒度上评估多轮LLM智能体服务策略,在CPU上以6%误差复现真实系统行为。

摘要:

多轮LLM智能体将模型调用与外部工具调用交织在一起,将服务从无状态请求处理转变为有状态程序执行。处理这些工作负载需要利用程序级上下文的调度、KV缓存管理和路由策略,包括轮次依赖、工具引入的间隙和可重用的KV状态。直接在真实系统上评估此类策略成本高昂,因为每个设计点可能需要跨到达率、模型规模、服务实例数量和内存层次结构的专用加速器时间。模拟提供了一种可扩展的替代方案,但现有的LLM服务模拟器针对无状态请求级工作负载,因此忽略了智能体服务的核心动态:多轮程序执行、跨轮缓存局部性以及工具间隙期间的KV缓存驻留。我们提出了AGENTSERVESIM,一种面向多轮LLM智能体服务的硬件感知模拟器。AGENTSERVESIM通过可组合模块在程序粒度上评估服务策略:程序编排器保留程序标识和轮次顺序,工具模拟器实现工具引入的间隙,会话感知路由器维护程序到实例的亲和性以实现缓存感知调度,KV驻留模型跟踪策略定义的跨HBM、主机DRAM/CXL和驱逐的KV放置。在真实服务部署和硬件配置上,AGENTSERVESIM在关键性能指标上的误差在6%以内,且完全在普通CPU上运行。这些结果表明,AGENTSERVESIM能够在不需在昂贵加速器上全面部署的情况下,实现受控、可重复的智能体服务策略探索。

英文摘要:

Large language model agents execute programs comprising multiple model turns interleaved with external tool calls. Their job completion time depends on how the serving system retains KV state across tool gaps, routes successor turns, and schedules competing programs. Most existing serving simulators operate on request streams in which arrivals are externally supplied and KV state follows request- or cache-scoped semantics. They therefore cannot jointly represent the cross-turn state and policy-dependent successor releases needed to evaluate counterfactual agent-serving trajectories. We present AgentServeSim, a simulator whose unit of execution is the agent program. A Program Control Block maintains cross-turn state, while a Program Orchestrator causally releases successor turns from simulated predecessor completions. A Retention Plane controls KV state across tool gaps, and a Dispatch Plane determines where and when each ready turn executes. We validate AgentServeSim against real vLLM deployments in 20 paired simulator-real cells spanning two GPU platforms, Llama-3.1-8B and Llama-3.1-70B, coding and function-calling agents, and five arrival rates. Mean JCT error remains within 5.5% on B200 and 5.2% in the saturated RTX PRO 6000 regime. Finally, we propose LLM-driven automated agent-serving policy search using AgentServeSim as a CPU-based fitness evaluator. The resulting policies improve mean JCT over hand-written seed policies by 0.5% for KV retention and 2.8% for scheduling.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑