DuplexMamba：通过双工和流式能力增强实时语音对话
DuplexMamba: Enhancing Real-time Speech Conversations with Duplex and Streaming Capabilities
AI总结：
本文提出DuplexMamba，一种基于Mamba的端到端多模态双工模型，通过同时处理输入和生成输出的双工解码策略实现实时流式语音对话，在ASR和语音助手基准上性能媲美Transformer模型。
摘要：
实时语音对话对于自然高效的人机交互至关重要，需要双工和流式能力。传统的基于Transformer的对话聊天机器人以轮次方式进行操作，并表现出随着输入规模增加而呈二次方增长的计算复杂度。在本文中，我们提出了DuplexMamba，一种基于Mamba的端到端多模态双工模型，用于语音到文本的对话。DuplexMamba能够同时进行输入处理和输出生成，并动态调整以支持实时流式传输。具体而言，我们开发了一个基于Mamba的语音编码器，并将其与基于Mamba的语言模型相适应。此外，我们引入了一种新颖的双工解码策略，使DuplexMamba能够同时处理输入和生成输出。实验结果表明，DuplexMamba成功实现了双工和流式能力，同时在自动语音识别（ASR）任务和语音助手基准评估中取得了与最近开发的几种基于Transformer的模型相当的性能。我们的代码和模型已发布。
英文摘要：
Real-time speech conversation is essential for natural and efficient human-machine interactions, requiring duplex and streaming capabilities. Traditional Transformer-based conversational chatbots operate in a turn-based manner and exhibit quadratic computational complexity that grows as the input size increases. In this paper, we propose DuplexMamba, a Mamba-based end-to-end multimodal duplex model for speech-to-text conversation. DuplexMamba enables simultaneous input processing and output generation, dynamically adjusting to support real-time streaming. Specifically, we develop a Mamba-based speech encoder and adapt it with a Mamba-based language model. Furthermore, we introduce a novel duplex decoding strategy that enables DuplexMamba to process input and generate output simultaneously. Experimental results demonstrate that DuplexMamba successfully implements duplex and streaming capabilities while achieving performance comparable to several recently developed Transformer-based models in automatic speech recognition (ASR) tasks and voice assistant benchmark evaluations. Our code and model are released.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
