连续令牌扩散用于多模态大语言模型中的说话人参考文本到语音合成
Continuous-Token Diffusion for Speaker-Referenced TTS in Multimodal LLMs
AI总结：
针对多模态大语言模型中TTS任务，提出连续令牌扩散与双头架构，结合两阶段训练，在LibriSpeech上实现SOTA自回归性能，显著降低WER。
摘要：
多模态大语言模型（MLLM）中的统一架构在单一框架内处理多种任务方面展现出前景。在文本到语音（TTS）任务中，当前基于MLLM的方法依赖离散令牌表示，这忽视了语音固有的连续性质，可能导致细粒度声学信息的丢失。在本工作中，我们研究了MLLM范式下使用连续语音表示的TTS。我们设计了一个双头架构，并实现了两种互补的训练策略以构建稳健的模型。（1）在MLLM上添加一个生成连续语音表示的扩散头，该扩散头是帧级别的且严格自回归的。（2）保留原始语言模型头以维持多任务能力并控制语音合成的开始和结束。（3）采用掩码训练来解决自回归解码中的暴露偏差。（4）为稳定优化，我们提出了一种两阶段方案，其中在第二阶段冻结语言模型，确保扩散头从固定的输入分布中学习。在LibriSpeech(PC) test-clean上的评估表明，我们的方法实现了最先进的自回归性能，词错误率（WER）为1.95%，说话人相似度为0.54，UTMOS为4.00。两阶段训练相比单阶段训练基线，相对WER降低了46%。这些结果凸显了将自回归建模与连续令牌扩散相结合，并辅以两阶段训练过程的有效性。
英文摘要：
Unified architectures in multimodal large language models (MLLM) have shown promise in handling diverse tasks within a single framework. In the text-to-speech (TTS) task, current MLLM-based approaches rely on discrete token representations, which disregard the inherently continuous nature of speech and can lead to loss of fine-grained acoustic information. In this work, we investigate the TTS within the MLLM paradigm using continuous speech representations. We design a dual-head architecture and implement two complementary training strategies for a robust model. (1) A diffusion head generating continuous speech representations is added on the MLLM, which is on frame-level and strictly autoregressive. (2) The original language model head is retained to preserve multitask capability and to control the start and end of speech synthesis. (3) Masked training is employed to address exposure bias in autoregressive decoding. (4) To stabilize optimization, we propose a two-stage scheme where the LM is frozen in the second stage, ensuring the diffusion head learns from a fixed input distribution. Evaluations on LibriSpeech(PC) test-clean show that our approach achieves state-of-the-art autoregressive performance, with a WER of 1.95%, speaker similarity of 0.54, and UTMOS of 4.00. The two-stage training yields a 46% relative WER reduction over the one-stage training baseline. These results highlight the effectiveness of combining autoregressive modeling with continuous-token diffusion, supported by a two-stage training procedure.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
