Stream RAG：具备流式工具使用能力的即时精准口语对话系统
Stream RAG: Instant and Accurate Spoken Dialogue Systems with Streaming Tool Usage
AI总结：
针对端到端语音对话系统易幻觉、工具集成延迟高的问题，提出流式RAG框架，通过并行预测工具查询降低延迟，构建AudioCRAG基准验证其可大幅提升问答准确率并减少工具使用延迟，且支持多模态输入。
摘要：
端到端语音输入语音输出对话系统正成为传统ASR-LLM-TTS流水线的有力替代方案，能生成更自然、更具表现力的回复，且延迟显著更低。然而，这类系统因事实基础有限，仍容易产生幻觉问题。虽然文本对话系统通过集成网页搜索、知识图谱API等工具应对了这一挑战，但我们提出了首个将工具使用直接拓展到语音输入语音输出系统的方法。一个核心挑战是，工具集成会大幅增加响应延迟，破坏对话流畅性。为缓解该问题，我们提出流式检索增强生成（Streaming RAG）——这一全新框架通过在用户语音输入的同时并行预测工具查询，甚至在用户尚未说完时就启动预测，来降低用户感知到的延迟。具体而言，我们开发了一套后训练流水线，用于教导模型在语音输入过程中何时发起工具调用，以及如何生成融合音频查询与检索到的文本结果的语音摘要，从而同时提升准确性与响应速度。为评估我们的方法，我们构建了AudioCRAG基准，它是将公开可用的CRAG数据集中的查询转换为语音形式得到的。实验结果表明，我们的流式RAG方法将问答准确率相对提升了最高200%（绝对值从11.1%提升至34.2%），还通过将工具使用延迟降低20%进一步优化了用户体验。重要的是，我们的流式RAG方法与模态无关，可同等应用于文本输入，为更具智能体能力的实时AI助手奠定了基础。
英文摘要：
End-to-end speech-in speech-out dialogue systems are emerging as a powerful alternative to traditional ASR-LLM-TTS pipelines, generating more natural, expressive responses with significantly lower latency. However, these systems remain prone to hallucinations due to limited factual grounding. While text-based dialogue systems address this challenge by integrating tools such as web search and knowledge graph APIs, we introduce the first approach to extend tool use directly into speech-in speech-out systems. A key challenge is that tool integration substantially increases response latency, disrupting conversational flow. To mitigate this, we propose Streaming Retrieval-Augmented Generation (Streaming RAG), a novel framework that reduces user-perceived latency by predicting tool queries in parallel with user speech, even before the user finishes speaking. Specifically, we develop a post-training pipeline that teaches the model when to issue tool calls during ongoing speech and how to generate spoken summaries that fuse audio queries with retrieved text results, thereby improving both accuracy and responsiveness. To evaluate our approach, we construct AudioCRAG, a benchmark created by converting queries from the publicly available CRAG dataset into speech form. Experimental results demonstrate that our streaming RAG approach increases QA accuracy by up to 200% relative (from 11.1% to 34.2% absolute) and further enhances user experience by reducing tool use latency by 20%. Importantly, our streaming RAG approach is modality-agnostic and can be applied equally to typed input, paving the way for more agentic, real-time AI assistants.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
