基于GMM的MDN音素级韵律建模用于多样化可控语音合成
Phone-Level Prosody Modelling with GMM-Based MDN for Diverse and Controllable Speech Synthesis
AI总结：
针对音素级韵律建模单峰分布导致语音多样性不足的问题，提出基于GMM的MDN建模方法并扩展至多说话人TTS，还实现参考语音韵律克隆，实验验证其在多样性、自然度及说话人相似度上的优势。
摘要：
生成具有多样且平滑韵律模式的自然语音是一项具有挑战性的任务。尽管已有研究采用音素级韵律分布的随机采样来生成不同的韵律模式，但生成语音的多样性仍然非常有限，远不及人类所能达到的水平。这主要是因为现有音素级韵律建模工作使用了单峰分布，例如单一高斯分布。在本研究中，我们提出了一种新方法，采用基于高斯混合模型（GMM）的混合密度网络（MDN）对音素级韵律进行建模，并通过对高斯均值和方差进行说话人自适应变换，将其扩展到多说话人TTS（文本转语音）场景。此外，我们证明可以通过从生成参考韵律的高斯分量中采样韵律，来克隆参考语音的韵律。我们在LJSpeech和LibriTTS数据集上的实验表明，基于GMM的MDN所提出的方法，在单说话人和多说话人TTS中，不仅多样性显著优于使用单一高斯的方法，自然度也更优。韵律克隆实验证明，该基于GMM的MDN方法的韵律相似度与近期提出的细粒度VAE相当，且目标说话人相似度更优。
英文摘要：
Generating natural speech with a diverse and smooth prosody pattern is a challenging task. Although random sampling with phone-level prosody distribution has been investigated to generate different prosody patterns, the diversity of the generated speech is still very limited and far from what can be achieved by humans. This is largely due to the use of uni-modal distribution, such as single Gaussian, in the prior works of phone-level prosody modelling. In this work, we propose a novel approach that models phone-level prosodies with a GMM-based mixture density network(MDN) and then extend it for multi-speaker TTS using speaker adaptation transforms of Gaussian means and variances. Furthermore, we show that we can clone the prosodies from a reference speech by sampling prosodies from the Gaussian components that produce the reference prosodies. Our experiments on LJSpeech and LibriTTS dataset show that the proposed method with GMM-based MDN not only achieves significantly better diversity than using a single Gaussian in both single-speaker and multi-speaker TTS, but also provides better naturalness. The prosody cloning experiments demonstrate that the prosody similarity of the proposed method with GMM-based MDN is comparable to recent proposed fine-grained VAE while the target speaker similarity is better.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
