ELLA-V:基于对齐引导序列重排的稳定神经编解码语言建模
ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reordering
AI总结:
本文提出ELLA-V,一种基于语言模型的零样本TTS框架,通过将音素token置于对应声学token之前的交错序列重排实现音素级细粒度控制,在准确率和稳定性上优于VALL-E。
摘要:
基于声学和语言提示的语言模型(LM)方法,如VALL-E,在零样本音频生成领域取得了显著进展。然而,现有方法仍存在一些局限:1)由于音频与音素token之间的对齐约束有限,合成语音中存在重复、换位和遗漏现象;2)使用自回归(AR)语言模型对合成语音进行细粒度控制面临挑战;3)由于AR解码的特性,尤其是在贪心策略下,会产生无限静音。为缓解这些问题,我们提出ELLA-V,一个简单但高效的基于LM的零样本文本到语音(TTS)框架,能够在音素级别对合成音频进行细粒度控制。ELLA-V的关键在于交错排列声学token和音素token序列,其中音素token出现在对应声学token之前。实验结果表明,我们的模型在准确率上优于VALL-E,并且在使用贪心解码和基于采样的解码策略时均能提供更稳定的结果。ELLA-V的代码将在清理后开源。音频样本可在https://ereboas.github.io/ELLAV/获取。
英文摘要:
The language model (LM) approach based on acoustic and linguistic prompts, such as VALL-E, has achieved remarkable progress in the field of zero-shot audio generation. However, existing methods still have some limitations: 1) repetitions, transpositions, and omissions in the output synthesized speech due to limited alignment constraints between audio and phoneme tokens; 2) challenges of fine-grained control over the synthesized speech with autoregressive (AR) language model; 3) infinite silence generation due to the nature of AR-based decoding, especially under the greedy strategy. To alleviate these issues, we propose ELLA-V, a simple but efficient LM-based zero-shot text-to-speech (TTS) framework, which enables fine-grained control over synthesized audio at the phoneme level. The key to ELLA-V is interleaving sequences of acoustic and phoneme tokens, where phoneme tokens appear ahead of the corresponding acoustic tokens. The experimental findings reveal that our model outperforms VALL-E in terms of accuracy and delivers more stable results using both greedy and sampling-based decoding strategies. The code of ELLA-V will be open-sourced after cleanups. Audio samples are available at https://ereboas.github.io/ELLAV/.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。