MPE-TTS：使用多模态提示的定制情感零样本文本到语音合成
MPE-TTS: Customized Emotion Zero-Shot Text-To-Speech Using Multi-Modal Prompt
AI总结：
提出基于多模态提示的定制情感零样本语音合成系统，解耦语音内容、音色、情感和韵律，通过多模态情感编码器和韵律预测器提升自然度与相似性。
摘要：
大多数现有的零样本文本到语音合成（ZS-TTS）系统基于单一提示（如参考语音或文本描述）生成未见过的语音，这限制了它们的灵活性。我们提出了一种基于多模态提示的定制情感ZS-TTS系统。该系统将语音解耦为内容、音色、情感和韵律，使得情感提示可以以文本、图像或语音的形式提供。为了从不同的提示中提取情感信息，我们提出了一种多模态提示情感编码器。此外，我们引入了一个韵律预测器来拟合韵律的分布，并提出了一个情感一致性损失，以在预测的韵律中保留情感信息。采用基于扩散的声学模型来生成目标梅尔频谱图。客观和主观实验均表明，我们的系统在自然度和相似性方面优于现有系统。样本可在https://mpetts-demo.github.io/mpetts_demo/获取。
英文摘要：
Most existing Zero-Shot Text-To-Speech(ZS-TTS) systems generate the unseen speech based on single prompt, such as reference speech or text descriptions, which limits their flexibility. We propose a customized emotion ZS-TTS system based on multi-modal prompt. The system disentangles speech into the content, timbre, emotion and prosody, allowing emotion prompts to be provided as text, image or speech. To extract emotion information from different prompts, we propose a multi-modal prompt emotion encoder. Additionally, we introduce an prosody predictor to fit the distribution of prosody and propose an emotion consistency loss to preserve emotion information in the predicted prosody. A diffusion-based acoustic model is employed to generate the target mel-spectrogram. Both objective and subjective experiments demonstrate that our system outperforms existing systems in terms of naturalness and similarity. The samples are available at https://mpetts-demo.github.io/mpetts_demo/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
