arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.26289cs.LG
收藏

面向低延迟多智能体工具调用的有状态推理

Stateful Inference for Low-Latency Multi-Agent Tool Calling

LayerScale, Inc.(LayerScale公司)

Victor Norgren

AI总结:

提出一种有状态推理架构,通过持久化KV缓存和增量处理,将多智能体工具调用的每轮成本从O(n_t)降至O(Δ_t),在6轮和35轮工作流中分别实现2.1倍和4.2倍的加速。

摘要:

多智能体工具调用正成为基于LLM系统的主要交互模式,但现有推理框架将每次工具调用视为独立请求,从头重新处理整个对话,尽管85-95%的提示与上一轮相同。我们提出一种有状态推理架构,将传统服务的每轮O(n_t)成本转换为仅增量O(Δ_t)成本:持久KV缓存跨轮次存在,仅通过摄入新令牌前进,而基数前缀缓存将其扩展到交错的多智能体流量,提示查找推测解码器加速结构化输出。在针对新颖、完全生成的工作负载的测试中,与vLLM和SGLang相比,参考实现在6轮智能体工作流中每轮快2.1倍,在35轮工作流的中位数轮次中快4.2倍,端到端挂钟时间减半。优势来自有状态重用和推测,而非缓存。

英文摘要:

Multi-agent tool calling is becoming the dominant interaction pattern for LLM-based systems, yet existing inference frameworks treat each tool call as an independent request, re-processing the entire conversation from scratch even though 85-95% of the prompt is unchanged from the previous turn. We present a stateful inference architecture that converts the $O(n_t)$ per-turn cost of conventional serving into an $O(Δ_t)$ delta-only cost: a persistent KV cache lives across turns and advances by ingesting only the new tokens, while a radix prefix cache extends this across interleaved multi-agent traffic and a prompt-lookup speculative decoder accelerates structured output. Against vLLM and SGLang on novel, fully-generated workloads, the reference implementation is $2.1\times$ faster per turn on a 6-turn agentic workflow and $4.2\times$ on the median turn of a 35-turn one, halving end-to-end wall time. The advantage comes from stateful reuse and speculation, not caching.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑