推测交互代理:构建具有异步I/O和推测性工具调用的实时代理
Speculative Interaction Agents: Building Real-Time Agents with Asynchronous I/O and Speculative Tool Calling
AI总结:
本文提出推测交互代理,通过异步I/O和推测性工具调用,实现复杂多轮工具调用的实时交互,提升实时应用性能。
摘要:
对于需要与人交互的应用,实时低延迟响应至关重要,例如语音控制应用通常要求1秒以内的延迟。然而,若LLM需推理并执行带有工具调用的代理工作流,这会增加数秒甚至更长的延迟,阻碍实时敏感应用。本文提出推测交互代理,通过异步I/O解耦核心代理推理与执行线程,使其在等待外部延迟时能重叠处理。同时提出推测性工具调用,用于管理任务执行,当代理不确定是否已收到完整信息或是否后续会提供更多信息时。对于强云模型,该方法可直接应用于现有实时云API,提供1.3-1.7倍的速度提升,伴随小幅精度损失。为实现实时交互,还提出基于时钟的训练方法,使模型适应流式输入和异步响应,并展示SFT的合成数据生成策略。整体方法在多个工具调用基准上提供1.6-2.2倍的速度提升。
英文摘要:
There is a growing demand for agentic AI technologies for a range of downstream applications like customer service and personal assistants. For applications where the agent needs to interact with a person, real-time low-latency responsiveness is required; for example, with voice-controlled applications, under 1 second of latency is typically required for the interaction to feel seamless. However, if we want the LLM to reason and execute an agentic workflow with tool calling, this can add several seconds or more of latency, which is prohibitive for real-time latency-sensitive applications. In our work, we propose Speculative Interaction Agents to enable real-time interaction even for agents with complex multi-turn tool calling. We propose Asynchronous I/O, which decouples the core agent reason-and-act thread from waiting for additional information from either the user or environment, thereby allowing for overlapping agentic processing while waiting on external delays. We also propose Speculative Tool Calling as a method to manage task execution when the agent is still unsure if it has received the full information or if additional user information may later be provided. For strong cloud models, our method can be applied out-of-the-box to existing real-time cloud APIs, providing 1.3-1.7$\times$ speedups with minor accuracy loss. To enable real-time interaction with small edge-scale models, we also present a clock-based training methodology that adapts the model to handle streaming inputs and asynchronous responses, and demonstrate a synthetic data generation strategy for SFT. Altogether, this approach provides 1.6-2.2$\times$ speedups with the Qwen2.5-3B-Instruct and Llama-3.2-3B-Instruct models across multiple tool calling benchmarks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。