arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.12967cs.DC
收藏

Sutradhara: 一种智能编排-引擎协同设计用于基于工具的代理推理

Sutradhara: An Intelligent Orchestrator-Engine Co-design for Tool-based Agentic Inference

Microsoft Research India(微软研究院印度);Microsoft M365 Research(微软M365研究)

Anish Biswas, Kanishk Goel, Srivarshinee S, Jayashree Mohan, Alind Khare, Anjaly Parayil, Ramachandran Ramjee, Chetan Bansal

AI总结:

Sutradhara通过整合编排与LLM服务,优化工具执行与LLM预填充的重叠,提升代理推理系统的吞吐量与延迟平衡,减少端到端延迟。

摘要:

代理应用是通过迭代调用外部工具完成复杂任务的LLM。此类基于工具的代理正成为部署语言模型的主导范式。不同于传统单轮推理,代理工作负载在生成最终响应前会链式调用多个LLM调用和工具执行,从而产生新的性能瓶颈,表现为最终答案的首词渲染(FTR)延迟增加。通过分析大规模请求,我们揭示了三个关键挑战:工具调用占FTR延迟的30-85%,尽管在迭代中存在大量上下文重用,KV缓存命中率却下降,且顺序编排浪费了请求内的并行性。这些瓶颈源于编排器和LLM引擎作为解耦黑箱的设计缺口。我们提出了Sutradhara,一种协同设计的代理推理系统,通过薄API将编排与LLM服务整合,实现三种优化:利用工具感知提示分割重叠工具执行与后续LLM预填充,流式执行工具以在解码期间逐步调度工具而非等待完整输出,以及编排器感知的缓存管理,利用语义提示提高命中率并减少缓存抖动。在vLLM上实现后,Sutradhara提升了代理系统的吞吐量-延迟权衡,能够在相同中位FTR延迟下支持高达77%的负载,或在相同负载下将中位FTR延迟降低15%,同时将端到端延迟降低11%。

英文摘要:

Agentic applications are LLMs that iteratively invoke external tools to accomplish complex tasks. Such tool-based agents are rapidly becoming the dominant paradigm for deploying language models in production. Unlike traditional single-turn inference, agentic workloads chain together multiple LLM calls and tool executions before producing a final response, creating a new performance bottleneck that manifests as increased latency in First Token Rendered (FTR) of the final answer. Through analysis of requests at production scale, we reveal three critical challenges: tool calls account for 30-85% of FTR latency, KV cache hit rates collapse despite substantial context reuse across iterations, and sequential orchestration wastes potential intra-request parallelism. These bottlenecks stem from a design gap in which orchestrators and LLM engines operate as decoupled black boxes, preventing cross-layer optimizations. We present Sutradhara, a co-designed agentic inference system that integrates orchestration with LLM serving through a thin API enabling three optimizations: overlap tool execution with subsequent LLM prefill using tool-aware prompt splitting, streaming tool execution to dispatch tools incrementally during decode rather than waiting for complete output, and orchestrator-aware cache management that uses semantic hints to improve hit rates and reduce thrashing. Implemented on vLLM, Sutradhara improves the throughput-latency trade-off in agentic systems, sustains up to 77% higher load at the same median FTR latency, or reduces median FTR latency by up to 15% at the same load while reducing end-to-end latency by upto 11% on A100 GPUs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑