无向量量化的自回归语音合成
Autoregressive Speech Synthesis without Vector Quantization
AI总结：
本文提出MELLE，一种无需向量量化的自回归语音合成方法，通过回归损失和变分推断直接生成连续梅尔频谱，相比VALL-E更简化且性能更优。
摘要：
我们提出了MELLE，一种基于连续值令牌的新型语言建模方法，用于文本到语音合成（TTS）。MELLE直接从文本条件自回归地生成连续的梅尔频谱图帧，绕过了向量量化的需求，而向量量化通常是为音频压缩设计的，与连续表示相比会牺牲保真度。具体而言，（i）我们不用交叉熵损失，而是应用回归损失并引入一种提出的频谱通量损失函数，来建模连续值令牌的概率分布；（ii）我们将变分推断整合到MELLE中，以促进采样机制，从而增强输出多样性和模型鲁棒性。实验表明，与两阶段编解码器语言模型VALL-E及其变体相比，单阶段的MELLE通过避免采样向量量化码的固有缺陷，缓解了鲁棒性问题，在多个指标上取得了更优的性能，并且最重要的是，提供了一种更简化的范式。我们工作的演示可在https://aka.ms/melle获取。
英文摘要：
We present MELLE, a novel continuous-valued token based language modeling approach for text-to-speech synthesis (TTS). MELLE autoregressively generates continuous mel-spectrogram frames directly from text condition, bypassing the need for vector quantization, which is typically designed for audio compression and sacrifices fidelity compared to continuous representations. Specifically, (i) instead of cross-entropy loss, we apply regression loss with a proposed spectrogram flux loss function to model the probability distribution of the continuous-valued tokens; (ii) we have incorporated variational inference into MELLE to facilitate sampling mechanisms, thereby enhancing the output diversity and model robustness. Experiments demonstrate that, compared to the two-stage codec language model VALL-E and its variants, the single-stage MELLE mitigates robustness issues by avoiding the inherent flaws of sampling vector-quantized codes, achieves superior performance across multiple metrics, and, most importantly, offers a more streamlined paradigm. The demos of our work are provided at https://aka.ms/melle.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
