从粗到细的语音生成：通过多尺度语音编码与生成改进神经编解码器语言模型
Speaking from Coarse to Fine: Improving Neural Codec Language Model via Multi-Scale Speech Coding and Generation
AI总结：
针对神经编解码器语言模型的近因偏差问题，提出由粗到细的CoFi-Speech方法，通过多尺度编解码与两种生成模式，显著提升零样本TTS的语音自然度与说话人相似度。
摘要：
神经编解码器语言模型（CLM）已在文本转语音（TTS）合成中展现出卓越性能。然而，受“近因偏差”困扰，CLM对更高时间尺度的粗粒度信息缺乏足够关注，往往生成不自然甚至难以理解的语音。本研究提出CoFi-Speech，一种由粗到细的CLM-TTS方法，采用多尺度语音编码与生成来解决这一问题。我们训练了一个多尺度神经编解码器CoFi-Codec，将语音编码为多尺度离散表示，该表示由多个具有不同时间分辨率的token序列组成。随后，我们提出CoFi-LM，可通过两种模式生成该表示：基于单LM的尺度链生成，以及基于多LM的尺度栈生成。实验中，CoFi-Speech在零样本TTS的自然度和说话人相似度上显著优于单尺度基线系统。多尺度编码分析表明，CoFi-Codec在学习多尺度离散语音表示的同时，能保持高质量的语音重建，验证了其有效性。由粗到细的多尺度生成，尤其是尺度栈方法，也被证实是构建高质量TTS神经编解码器语言模型的关键途径。
英文摘要：
The neural codec language model (CLM) has demonstrated remarkable performance in text-to-speech (TTS) synthesis. However, troubled by ``recency bias", CLM lacks sufficient attention to coarse-grained information at a higher temporal scale, often producing unnatural or even unintelligible speech. This work proposes CoFi-Speech, a coarse-to-fine CLM-TTS approach, employing multi-scale speech coding and generation to address this issue. We train a multi-scale neural codec, CoFi-Codec, to encode speech into a multi-scale discrete representation, comprising multiple token sequences with different time resolutions. Then, we propose CoFi-LM that can generate this representation in two modes: the single-LM-based chain-of-scale generation and the multiple-LM-based stack-of-scale generation. In experiments, CoFi-Speech significantly outperforms single-scale baseline systems on naturalness and speaker similarity in zero-shot TTS. The analysis of multi-scale coding demonstrates the effectiveness of CoFi-Codec in learning multi-scale discrete speech representations while keeping high-quality speech reconstruction. The coarse-to-fine multi-scale generation, especially for the stack-of-scale approach, is also validated as a crucial approach in pursuing a high-quality neural codec language model for TTS.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
