SALMONN-omni：一种无需码本注入的独立全双工语音大语言模型
SALMONN-omni: A Standalone Speech LLM without Codec Injection for Full-duplex Conversation
AI总结：
SALMONN-omni是首个无需音频码本注入的独立全双工语音大语言模型，通过动态思考机制实现说话与聆听状态切换，在口语问答和开放域对话基准上相对现有开源全双工模型提升至少30%，并有效处理插话和回声消除等复杂场景。
摘要：
为了实现流畅自然的人机语音交互，现有的全双工对话系统通常采用模块化架构，并配备语音活动检测器、中断器、对话状态预测器或多个大语言模型等辅助组件。然而，这些系统存在模块间错误累积的问题，并难以应对上下文相关的插话（barge-in）和回声消除等关键挑战。近期的方法，尤其是Moshi，通过将音频码本注入到单个大语言模型的令牌空间中，简化了处理流程。然而，此类方法在语音而非文本模态上运行时，仍会带来显著的性能下降。在本文中，我们提出了SALMONN-omni，这是首个在令牌空间中无需音频码本的单一独立全双工语音大语言模型。它在模型主干中引入了一种新颖的动态思考机制，使模型能够学习何时在说话和聆听状态之间进行切换。在广泛使用的口语问答和开放域对话基准上的实验表明，与现有的开源全双工模型相比，SALMONN-omni实现了至少30%的相对性能提升，并且尽管使用的训练数据大幅减少，其性能仍与半双工和轮次式系统高度竞争。此外，SALMONN-omni在复杂对话场景中展现出强大性能，包括话轮转换、反馈信号、回声消除和上下文相关的插话，并通过强化学习获得了进一步提升。用户与SALMONN-omni之间的一些演示对话可在以下仓库中获取：https://github.com/bytedance/SALMONN。
英文摘要：
In order to enable fluid and natural human-machine speech interaction, existing full-duplex conversational systems often adopt modular architectures with auxiliary components such as voice activity detectors, interrupters, conversation state predictors, or multiple LLMs. These systems, however, suffer from error accumulation across modules and struggle with key challenges such as context-dependent barge-in and echo cancellation. Recent approaches, most notably Moshi, simplify the pipeline by injecting audio codecs into the token space of a single LLM. However, such methods still incur significant performance degradation when operating on the speech rather than text modality. In this paper, we introduce SALMONN-omni, the first single, standalone full-duplex speech LLM that operates without audio codecs in its token space. It features a novel dynamic thinking mechanism within the LLM backbone, enabling the model to learn when to transition between speaking and listening states. Experiments on widely used benchmarks for spoken question answering and open-domain dialogue show that SALMONN-omni achieves at least 30\% relative performance improvement over existing open-source full-duplex models and performs highly competitively to half-duplex and turn-based systems, despite using substantially less training data. Moreover, SALMONN-omni demonstrates strong performance in complex conversational scenarios, including turn-taking, backchanneling, echo cancellation and context-dependent barge-in, with further improvements achieved through reinforcement learning. Some demo conversations between user and SALMONN-omni are provided in the following repository https://github.com/bytedance/SALMONN.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
