arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2505.15670cs.CLcs.SDeess.AS

SALM-Duplex：面向语音到语音语言模型的高效直接双工建模

SALM-Duplex: Efficient and Direct Duplex Modeling for Speech-to-Speech Language Model

NVIDIAUSA（NVIDIA美国）

Ke Hu, Ehsan Hosseini-Asl, Chen Chen, Edresson Casanova, Subhankar Ghosh, Piotr Żelasko, Zhehuai Chen, Jason Li, Jagadeesh Balam, Boris Ginsburg

备注：
Accepted to Interspeech 2025
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出SALM-Duplex双工S2S架构，通过通道融合、流式编码器和独立编解码器建模实现实时插话与低比特率语音对话，并公开训练推理代码。

摘要：

口语对话是一种直观的人机交互形式，但当前的语音语言模型往往仍局限于轮流交换，缺乏用户插话等实时适应能力。我们提出一种新颖的双工语音到语音（S2S）架构，其特征是连续的用户输入和编解码器智能体输出，并通过通道融合直接对同时存在的用户与智能体流进行建模。使用预训练的流式编码器处理用户输入，使该模型成为首个无需语音预训练的双工S2S模型。用于智能体和用户建模的独立架构便于进行编解码器微调，从而获得更好的智能体语音，并将比特率较先前工作减半（0.6 kbps）。实验结果表明，所提出的模型在推理、话轮转换和插话能力方面优于先前的双工模型。由于跳过了语音预训练，该模型所需的语音数据显著减少，这明显简化了从任意LLM构建双工S2S模型的过程。最后，它是首个公开提供并附带训练与推理代码的双工S2S模型，以促进可复现性。

英文摘要：

Spoken dialogue is an intuitive form of human-computer interaction, yet current speech language models often remain constrained to turn-based exchanges, lacking real-time adaptability such as user barge-in. We propose a novel duplex speech to speech (S2S) architecture featuring continuous user inputs and codec agent outputs with channel fusion that directly models simultaneous user and agent streams. Using a pretrained streaming encoder for user input enables the first duplex S2S model without requiring speech pretrain. Separate architectures for agent and user modeling facilitate codec fine-tuning for better agent voices and halve the bitrate (0.6 kbps) compared to previous works. Experimental results show that the proposed model outperforms previous duplex models in reasoning, turn-taking, and barge-in abilities. The model requires significantly less speech data, as speech pretrain is skipped, which markedly simplifies the process of building a duplex S2S model from any LLMs. Finally, it is the first openly available duplex S2S model with training and inference code to foster reproducibility.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑