Mega-TTS 2：增强零样本语音合成的提示机制
Mega-TTS 2: Boosting Prompting Mechanisms for Zero-Shot Speech Synthesis
AI总结：
本文提出Mega-TTS 2，通过解耦韵律与音色、多参考音色编码及P-LLM增强零样本TTS提示机制，在10秒至5分钟数据设置下持续优于微调，并支持可控风格迁移。
摘要：
零样本文本转语音（TTS）旨在利用未见过的语音提示合成声音，通过跳过微调过程，显著降低声音克隆所需的数据和计算要求。然而，零样本TTS的提示机制仍在以下方面面临挑战：1）以往的零样本TTS工作通常使用单句提示进行训练，当推理阶段数据相对充足时，这显著限制了其性能。2）提示中的韵律信息与音色高度耦合，导致二者无法相互迁移。本文介绍Mega-TTS 2，这是一种用于零样本TTS的通用提示机制，用以应对上述挑战。具体而言，我们设计了一个强大的声学自编码器，将韵律和音色信息分别编码到压缩的潜在空间中，同时提供高质量重建。随后，我们提出多参考音色编码器和韵律潜在语言模型（P-LLM），以从多句提示中提取有用信息。我们进一步利用来自多个P-LLM输出的概率，生成可迁移且可控的韵律。实验结果表明，Mega-TTS 2不仅能够利用来自任意来源的未见说话人的简短提示合成保持身份特征的语音，而且在数据量从10秒到5分钟不等时，始终优于微调方法。此外，我们的方法能够以细粒度且受控的方式将多种说话风格迁移到目标音色。音频样本可在https://boostprompt.github.io/boostprompt/获取。
英文摘要：
Zero-shot text-to-speech (TTS) aims to synthesize voices with unseen speech prompts, which significantly reduces the data and computation requirements for voice cloning by skipping the fine-tuning process. However, the prompting mechanisms of zero-shot TTS still face challenges in the following aspects: 1) previous works of zero-shot TTS are typically trained with single-sentence prompts, which significantly restricts their performance when the data is relatively sufficient during the inference stage. 2) The prosodic information in prompts is highly coupled with timbre, making it untransferable to each other. This paper introduces Mega-TTS 2, a generic prompting mechanism for zero-shot TTS, to tackle the aforementioned challenges. Specifically, we design a powerful acoustic autoencoder that separately encodes the prosody and timbre information into the compressed latent space while providing high-quality reconstructions. Then, we propose a multi-reference timbre encoder and a prosody latent language model (P-LLM) to extract useful information from multi-sentence prompts. We further leverage the probabilities derived from multiple P-LLM outputs to produce transferable and controllable prosody. Experimental results demonstrate that Mega-TTS 2 could not only synthesize identity-preserving speech with a short prompt of an unseen speaker from arbitrary sources but consistently outperform the fine-tuning method when the volume of data ranges from 10 seconds to 5 minutes. Furthermore, our method enables to transfer various speaking styles to the target timbre in a fine-grained and controlled manner. Audio samples can be found in https://boostprompt.github.io/boostprompt/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
