arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.08607cs.CLcs.SD
收藏

VocalNet-MDM:通过自蒸馏掩码扩散建模加速流式语音大语言模型

VocalNet-MDM: Accelerating Streaming Speech LLM via Self-Distilled Masked Diffusion Modeling

Shanghai Jiao Tong University(上海交通大学);Ant Group(蚂蚁集团)

Ziyang Cheng, Yuhao Wang, Heyang Liu, Ronghua Wu, Qunshan Gu, Yanfeng Wang, Yu Wang

AI总结:

VocalNet-MDM通过自蒸馏掩码扩散建模实现流式语音LLM的高效解码与低延迟

摘要:

近年来,语音大语言模型(LLMs)在端到端语音交互中实现了令人印象深刻的能力。然而,主流的自回归范式 imposes 严格的串行约束,限制了生成效率并引入了曝光偏差。在本文中,我们研究了掩码扩散建模(MDM)作为一种非自回归范式用于语音LLMs,并引入了VocalNet-MDM。为了将MDM适应于流式语音交互,我们解决了两个关键挑战:训练-推理不匹配和迭代开销。我们提出分层块级掩码以对齐训练目标与块扩散解码过程中遇到的逐步掩码状态,并提出迭代自蒸馏以将多步细化压缩到更少的步骤中以实现低延迟推理。在仅6000小时语音数据上训练,VocalNet-MDM实现了3.7倍至10倍的解码加速,并将第一块延迟降低了34% compared to AR基线。它在保持竞争性的识别准确性的同时,实现了最先进的文本质量和语音自然度,证明了MDM是低延迟、高效语音LLMs的有前景且可扩展的替代方案。

英文摘要:

Recent Speech Large Language Models~(LLMs) have achieved impressive capabilities in end-to-end speech interaction. However, the prevailing autoregressive paradigm imposes strict serial constraints, limiting generation efficiency and introducing exposure bias. In this paper, we investigate Masked Diffusion Modeling~(MDM) as a non-autoregressive paradigm for speech LLMs and introduce VocalNet-MDM. To adapt MDM for streaming speech interaction, we address two critical challenges: training-inference mismatch and iterative overhead. We propose Hierarchical Block-wise Masking to align training objectives with the progressive masked states encountered during block diffusion decoding, and Iterative Self-Distillation to compress multi-step refinement into fewer steps for low-latency inference. Trained on a limited scale of only 6K hours of speech data, VocalNet-MDM achieves a 3.7$\times$--10$\times$ decoding speedup and reduces first-chunk latency by 34\% compared to AR baselines. It maintains competitive recognition accuracy while achieving state-of-the-art text quality and speech naturalness, demonstrating that MDM is a promising and scalable alternative for low-latency, efficient speech LLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑