CLAPSpeech：基于对比语言-音频预训练从文本上下文中学习韵律
CLAPSpeech: Learning Prosody from Text Context with Contrastive Language-Audio Pre-training
AI总结：
CLAPSpeech提出跨模态对比预训练框架，显式学习文本令牌在不同上下文下的韵律变化，通过多尺度预训练流程捕获多层级韵律模式，提升TTS韵律预测并展现多语言与多说话人泛化能力。
摘要：
为了实现富有表现力的文本到语音（TTS），改进文本表示引起了广泛关注。然而，现有工作仅通过掩码令牌重建任务隐式地学习韵律，这导致训练效率低下且韵律建模困难。我们提出了 CLAPSpeech，这是一个跨模态对比预训练框架，能够显式地学习相同文本令牌在不同上下文下的韵律变化。具体而言：1）我们通过精心设计编码器输入和对比损失，鼓励模型在联合多模态空间中将文本上下文与其对应的韵律模式连接起来；2）我们引入了多尺度预训练流程，以在多个层级上捕获韵律模式。我们展示了如何将 CLAPSpeech 融入现有的 TTS 模型以获得更好的韵律。在三个数据集上的实验不仅表明 CLAPSpeech 能够改进现有 TTS 方法的韵律预测，还证明了其适应多语言和多说话人 TTS 的泛化能力。我们还深入分析了 CLAPSpeech 性能背后的原理。消融研究证明了我们方法中每个组件的必要性。源代码和音频样本可在 https://clapspeech.github.io 获取。
英文摘要：
Improving text representation has attracted much attention to achieve expressive text-to-speech (TTS). However, existing works only implicitly learn the prosody with masked token reconstruction tasks, which leads to low training efficiency and difficulty in prosody modeling. We propose CLAPSpeech, a cross-modal contrastive pre-training framework that explicitly learns the prosody variance of the same text token under different contexts. Specifically, 1) We encourage the model to connect the text context with its corresponding prosody pattern in the joint multi-modal space with the elaborate design of the encoder inputs and contrastive loss; 2) We introduce a multi-scale pre-training pipeline to capture prosody patterns in multiple levels. We show how to incorporate CLAPSpeech into existing TTS models for better prosody. Experiments on three datasets not only show that CLAPSpeech could improve the prosody prediction for existing TTS methods, but also demonstrate its generalization ability to adapt to multiple languages and multi-speaker TTS. We also deeply analyze the principle behind the performance of CLAPSpeech. Ablation studies demonstrate the necessity of each component in our method. Source code and audio samples are available at https://clapspeech.github.io.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
