SALMONN-omni：用于全双工语音理解与生成的无编解码器大语言模型
SALMONN-omni: A Codec-free LLM for Full-duplex Speech Understanding and Generation
AI总结：
针对全双工语音理解与生成任务，提出无编解码器的SALMONN-omni模型，通过双工口语对话框架及“思考”机制实现异步生成，在多任务中表现通用且能处理复杂对话场景。
摘要：
全双工多模态大语言模型（LLMs）为解决多样化的语音理解和生成任务提供了统一框架，能实现更自然流畅的人机对话。与将语音识别、理解和文本转语音生成拆分为独立组件的传统模块化对话AI系统不同，多模态LLMs以单一端到端模型运行。这种精简设计消除了组件间的错误传播，并充分利用了输入语音信号中嵌入的丰富非语言信息。我们提出SALMONN-omni，一种无编解码器的全双工语音理解与生成模型，能在说话时同时监听自身生成的语音和背景声音。为支持该能力，我们提出了一种新颖的双工口语对话框架，融入了“思考”机制，基于嵌入而非编解码器（量化语音和音频令牌）实现异步文本和语音生成。实验结果表明，SALMONN-omni在语音识别、语音增强和口语问答等广泛的流式语音任务中具有通用性，还能出色处理话轮转换、插话和回声消除场景，展现出作为稳健全双工对话AI系统原型的潜力。据我们所知，SALMONN-omni是同类首个无编解码器模型。完整技术报告和模型检查点即将发布。
英文摘要：
Full-duplex multimodal large language models (LLMs) provide a unified framework for addressing diverse speech understanding and generation tasks, enabling more natural and seamless human-machine conversations. Unlike traditional modularised conversational AI systems, which separate speech recognition, understanding, and text-to-speech generation into distinct components, multimodal LLMs operate as single end-to-end models. This streamlined design eliminates error propagation across components and fully leverages the rich non-verbal information embedded in input speech signals. We introduce SALMONN-omni, a codec-free, full-duplex speech understanding and generation model capable of simultaneously listening to its own generated speech and background sounds while speaking. To support this capability, we propose a novel duplex spoken dialogue framework incorporating a ``thinking'' mechanism that facilitates asynchronous text and speech generation relying on embeddings instead of codecs (quantized speech and audio tokens). Experimental results demonstrate SALMONN-omni's versatility across a broad range of streaming speech tasks, including speech recognition, speech enhancement, and spoken question answering. Additionally, SALMONN-omni excels at managing turn-taking, barge-in, and echo cancellation scenarios, establishing its potential as a robust prototype for full-duplex conversational AI systems. To the best of our knowledge, SALMONN-omni is the first codec-free model of its kind. A full technical report along with model checkpoints will be released soon.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
