面向零样本语音克隆的多模态对抗训练
Multi-modal Adversarial Training for Zero-Shot Voice Cloning
AI总结：
针对零样本语音克隆中TTS模型预测趋于平均化的问题，提出基于Transformer编码器-解码器的GAN判别器，结合FastSpeech2声学模型与Libriheavy数据集进行多模态对抗训练，显著提升了语音质量与说话人相似度。
摘要：
训练用于在给定文本下重建语音的文本到语音（TTS）模型，其预测往往倾向于接近数据集的平均特征，无法对使人类语音听起来自然的变异进行建模。对于零样本语音克隆这一需要高说话风格方差训练数据的任务，该问题被进一步放大。我们基于近期使用生成对抗网络（GAN）的研究，提出了一种Transformer编码器-解码器架构，以有条件地判别真实与生成的语音特征。该判别器被用于一个改进TTS模型声学和韵律特征的训练流程中。我们将这种新颖的对抗训练技术应用于FastSpeech2声学模型，并在大型多说话人数据集Libriheavy上进行训练，以完成零样本语音克隆任务。在语音质量和说话人相似度方面，我们的模型取得了优于基线的改进。系统的音频示例可在线获取。
英文摘要：
A text-to-speech (TTS) model trained to reconstruct speech given text tends towards predictions that are close to the average characteristics of a dataset, failing to model the variations that make human speech sound natural. This problem is magnified for zero-shot voice cloning, a task that requires training data with high variance in speaking styles. We build off of recent works which have used Generative Advsarial Networks (GAN) by proposing a Transformer encoder-decoder architecture to conditionally discriminates between real and generated speech features. The discriminator is used in a training pipeline that improves both the acoustic and prosodic features of a TTS model. We introduce our novel adversarial training technique by applying it to a FastSpeech2 acoustic model and training on Libriheavy, a large multi-speaker dataset, for the task of zero-shot voice cloning. Our model achieves improvements over the baseline in terms of speech quality and speaker similarity. Audio examples from our system are available online.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
