SoulX-Duplug:用于实时全双工语音对话的即插即用流式状态预测模块
SoulX-Duplug: Plug-and-Play Streaming State Prediction Module for Realtime Full-Duplex Speech Conversation
AI总结:
本文提出SoulX-Duplug模块,通过联合流式ASR利用文本信息识别用户意图,提升全双工语音对话的低延迟状态控制与轮次管理性能。
摘要:
近期 spoken dialogue systems 的进展引发了对人类样全双工语音交互的更多关注。然而,我们对该领域的全面回顾揭示了几个挑战,包括训练数据获取困难、灾难性遗忘和可扩展性有限。本文提出SoulX-Duplug,一种用于全双工语音对话系统的即插即用流式状态预测模块。通过联合执行流式ASR,SoulX-Duplug明确利用文本信息来识别用户意图,有效充当语义VAD。为促进公平评估,我们引入SoulX-Duplug-Eval,扩展广泛使用的基准测试,以提高双语覆盖范围。实验结果表明,SoulX-Duplug实现了低延迟的流式对话状态控制,基于其构建的系统在整体轮次管理和延迟性能上优于现有全双工模型。我们已开源SoulX-Duplug和SoulX-Duplug-Eval。
英文摘要:
Recent advances in spoken dialogue systems have brought increased attention to human-like full-duplex voice interactions. However, our comprehensive review of this field reveals several challenges, including the difficulty in obtaining training data, catastrophic forgetting, and limited scalability. In this work, we propose SoulX-Duplug, a plug-and-play streaming state prediction module for full-duplex spoken dialogue systems. By jointly performing streaming ASR, SoulX-Duplug explicitly leverages textual information to identify user intent, effectively serving as a semantic VAD. To promote fair evaluation, we introduce SoulX-Duplug-Eval, extending widely used benchmarks with improved bilingual coverage. Experimental results show that SoulX-Duplug enables low-latency streaming dialogue state control, and the system built upon it outperforms existing full-duplex models in overall turn management and latency performance. We have open-sourced SoulX-Duplug and SoulX-Duplug-Eval.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。