arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.10342cs.DC
收藏

AgentServe: 为在消费级GPU上高效执行代理AI服务的算法-系统协同设计

AgentServe: Algorithm-System Co-Design for Efficient Agentic AI Serving on a Consumer-Grade GPU

The University of Sydney(悉尼大学)

Yuning Zhang, Yan Yan, Nan Yang, Dong Yuan

AI总结:

AgentServe通过算法-系统协同设计,在消费级GPU上实现高效代理AI服务,提升延迟稳定性并保持吞吐量。

摘要:

大型语言模型(LLMs)越来越多地被部署为在短推理-行动循环中运行的AI代理,这些代理在模型计算和外部调用之间交替进行。与传统聊天应用不同,这些代理工作负载要求推理服务系统在多个来自不同AI代理的请求到达时,平衡低延迟、稳定的token生成和吞吐量。最近的部署显示,一种趋势是将小型语言模型(SLMs)在消费级GPU上本地运行,这受到隐私、合规性和成本限制的驱动。当异构请求在单个GPU上重叠时,长的prefill和短的decode争夺资源,产生头阻塞,导致交互性能不稳定。通过分析代理工作负载,我们发现其执行自然地分为冷prefill,处理长系统提示,恢复prefill,将工具输出附加到缓存的上下文中,以及短decode,这些是关键延迟。这种混合比传统聊天机器人服务更加紧张。我们提出了AgentServe,一个单GPU服务系统,通过隔离prefill和decode、对恢复prefill应用动态预算、并通过预建立的CUDA Green Context槽分配GPU资源来确保在这些条件下稳定的多代理执行。评估结果表明,AgentServe在保持竞争力吞吐量的同时显著提高了延迟稳定性,实现了在不同设置下相比现有最佳基线的2.8倍TTFT改进和2.7倍TPOT改进。

英文摘要:

Large language models (LLMs) are increasingly deployed as AI agents that operate in short reasoning-action loops, interleaving model computation with external calls. Unlike traditional chat applications, these agentic workloads require inference serving systems to balance low latency, stable token emission, and throughput under multiple request arrivals from different AI agents. Recent deployments highlight a shift toward running small language models (SLMs) locally on consumer-grade GPUs, driven by privacy, compliance, and cost constraints. When heterogeneous requests overlap on a single GPU, long prefills and short decodes contend for resources, creating head-of-line blocking that destabilizes interactive performance. By analyzing agent workloads, we observe that their execution naturally separates into cold prefills, which process long system prompts, resume prefills, which append tool outputs to cached contexts, and short decodes, which are latency-critical. This mix intensifies contention compared to conventional chatbot serving. We present AgentServe, a single-GPU serving system that ensures stable multi-agent execution under such conditions by isolating prefills from decodes, applying dynamic budgeting to resume prefills, and allocating GPU resources through pre-established CUDA Green Context slots with adaptive control. Evaluation results show that AgentServe significantly improves latency stability while sustaining competitive throughput, achieving up to 2.8x TTFT improvement and 2.7x TPOT improvement over state-of-the-art baselines across different settings.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑