DELTA-TTS：将自回归模型适配为扩散语言模型以实现文本转语音

中文摘要 AI 辅助

自回归（AR）文本转语音（TTS）模型按序生成离散语音令牌，存在推理速度慢的问题，且会因局部误差和幻觉的传播导致鲁棒性下降。该限制源于其从左到右的自回归约束：每个令牌必须在未来语音令牌上下文可用前确定。但此类排序并非TTS的固有要求，因为合成前完整输入文本已经可用。本文提出DELTA-TTS，一种基于LoRA的轻量适配框架，可将预训练AR TTS模型转换为离散扩散语言模型（dLLM），实现按置信度排序的语音令牌解码。为更好捕捉语音局部结构，DELTA-TTS集成了注入局部声学上下文的卷积模块，搭配$1/t$加权训练目标与时间偏移推理调度，将低置信度位置延后到后续步骤处理。仅在585小时LibriTTS数据集上训练后，DELTA-TTS在Seed-TTS test-en集上的WER达到1.75%，性能优于其AR骨干网络，同时令牌生成速度提升3.3倍。进一步分析表明，DELTA-TTS可实现更精准的文本-语音对齐，提升整体解码置信度，缓解AR生成过程中出现的幻觉问题。

英文摘要

Autoregressive (AR) text-to-speech (TTS) models generate discrete speech tokens sequentially, which makes inference slow and can degrade robustness, since local errors propagate to later positions and can escalate into hallucination. This limitation stems from their left-to-right AR commitment: each token must be determined before future speech-token context is available. However, such ordering is not an inherent requirement for TTS, since the model receives the full input text before synthesis. In this paper, we introduce DELTA-TTS, a lightweight LoRA-based adaptation framework that converts a pretrained AR TTS model into a discrete diffusion language model (dLLM) for confidence-ordered speech-token decoding. To better capture the local structure of speech, DELTA-TTS incorporates a convolution module that injects local acoustic context, together with a 1/t-weighted training objective and a time-shifted inference schedule that together defer low-confidence positions to later steps. Trained on only 585 hours of LibriTTS, DELTA-TTS achieves a 1.75% WER on Seed-TTS test-en, outperforming its AR backbone while generating tokens 3.3x faster. Further analysis shows that DELTA-TTS produces sharper text--speech alignment, increases overall decoding confidence, and mitigates the hallucinations observed in AR generation.

发表机构

• Sungkyunkwan University（首尔大学）
• University of Seoul（首尔大学）

机构由 AI 辅助整理，请以论文原文为准。

补充信息

备注

ICML 2026 SPIGM Workshop