MoshiRAG: 异步知识检索用于全双工语音语言模型
MoshiRAG: Asynchronous Knowledge Retrieval for Full-Duplex Speech Language Models
AI总结：
本文提出MoshiRAG，通过异步框架结合紧凑接口与选择性检索，提升全双工语音模型的事实性，同时保持交互性。
摘要：
语音到语音语言模型近年来涌现以增强对话式AI的自然性。特别是全双工模型以其实时交互性著称，包括处理停顿、打断和回声。然而，提升其事实性仍是一个开放挑战。尽管扩大模型规模可解决这一差距，但会使实时推理成本过高。本文提出MoshiRAG，一种模块化方法，结合紧凑的全双工接口与选择性检索以访问更强大的知识源。我们的异步框架使模型能够识别需要知识的查询并将其响应基于外部信息。通过利用响应开始与核心信息交付之间的自然时间间隙，检索过程可在保持自然对话流程的同时完成。采用此方法，MoshiRAG在事实性上可与最佳公开发布的非双工语音语言模型相媲美，同时保留全双工系统的交互性。此外，我们的灵活设计支持无需重新训练的插件式检索方法，并在跨领域数学推理任务上表现出色。
英文摘要：
Speech-to-speech language models have recently emerged to enhance the naturalness of conversational AI. In particular, full-duplex models are distinguished by their real-time interactivity, including handling of pauses, interruptions, and backchannels. However, improving their factuality remains an open challenge. While scaling the model size could address this gap, it would make real-time inference prohibitively expensive. In this work, we propose MoshiRAG, a modular approach that combines a compact full-duplex interface with selective retrieval to access more powerful knowledge sources. Our asynchronous framework enables the model to identify knowledge-demanding queries and ground its responses in external information. By leveraging the natural temporal gap between response onset and the delivery of core information, the retrieval process can be completed while maintaining a natural conversation flow. With this approach, MoshiRAG achieves factuality comparable to the best publicly released non-duplex speech language models while preserving the interactivity inherent to full-duplex systems. Moreover, our flexible design supports plug-and-play retrieval methods without retraining and demonstrates strong performance on out-of-domain mathematical reasoning tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
