arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.00528cs.DCcs.AIcs.LGcs.OS
收藏

SAGA:面向GPU集群的AI代理推理工作流原子调度

SAGA: Workflow-Atomic Scheduling for AI Agent Inference on GPU Clusters

The University of Hong Kong(香港大学);Brain Investing Limited(脑投有限公司);Stellaris AI Limited(Stellaris AI有限公司)

Dongxin Guo, Jikun Wu, Siu Ming Yiu

AI总结:

SAGA通过工作流级调度减少任务完成时间1.64倍,提升GPU内存利用率1.22倍,同时在多租户环境下达成99.2%的服务水平目标,但牺牲了30%的峰值吞吐量。

摘要:

AI代理每个任务执行数十到数百次链式LLM调用,但GPU调度器将每个调用视为独立任务,丢弃步骤间的数十GB中间状态,导致端到端延迟增加3-8倍。本文认为请求级抽象与复合AI工作负载不匹配,提出工作流级调度:将整个代理工作流(而非单个推理调用)作为可调度单元。SAGA通过三种机制实现:(1)代理执行图捕获工作流结构,预测工具调用边界间的KV缓存重用,达到Bélády最优离线策略的1.31倍;(2)会话亲和力分组配合工作窃取,协同相关请求同时保持全局负载平衡;(3)代理公平分享,一个具有证明有界偏差保证的任务完成时间公平度量。在64个GPU集群上服务SWE-bench编码代理和WebArena浏览器任务,SAGA在vLLM v0.15.1基础上减少任务完成时间1.64倍(几何均值,p < 0.001),提升GPU内存利用率1.22倍,达成99.2%的SLO目标,同时在多租户干扰下实现。这些延迟收益以约30%的峰值吞吐量降低为代价,这种权衡适用于主导复合AI使用的延迟敏感交互部署。

英文摘要:

AI agents execute tens to hundreds of chained LLM calls per task, yet GPU schedulers treat each call as independent, discarding gigabytes of intermediate state between steps and inflating end-to-end latency by 3-8x. We argue that this request-level abstraction is fundamentally mismatched to compound AI workloads, and propose a shift to program-level scheduling: treating the entire agent workflow (not individual inference calls) as the first-class schedulable unit. We present SAGA, a distributed scheduler that implements this abstraction through three mechanisms: (1) Agent Execution Graphs that capture workflow structure to predict KV cache reuse across tool-call boundaries, achieving within 1.31x of Bélády's optimal offline policy; (2) session-affinity batching with work stealing that co-locates correlated requests while maintaining global load balance; and (3) Agent Fair Share, a task-completion-time fairness metric with provable bounded-deviation guarantees. On a 64-GPU cluster serving SWE-bench coding agents and WebArena browser tasks, SAGA reduces task completion time by 1.64x (geometric mean, p < 0.001) over vLLM v0.15.1 with prefix caching and affinity routing, while improving GPU memory utilization by 1.22x and achieving 99.2% SLO attainment under multi-tenant interference. These latency gains come at a quantified cost: approximately 30% lower peak throughput than throughput-optimal batch scheduling, a tradeoff appropriate for the latency-sensitive interactive deployments that dominate compound AI usage. Our results demonstrate that workflow-aware scheduling is essential for efficient compound AI serving.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑