结合掩码语言建模与跨模态对比学习的语调感知语音合成
Combining Masked Language Modeling and Cross-Modal Contrastive Learning for Prosody-Aware TTS
AI总结：
本文研究了基于扩散模型的语音合成中多阶段预训练对语调建模的影响，通过掩码语言建模与混合音素对比学习相结合，提升了生成质量与感知指标。
摘要：
我们研究了基于扩散模型的语音合成中多阶段预训练对语调建模的影响。一个说话者条件的双流编码器通过掩码语言建模后，采用混合音素批次进行SigLIP风格的跨模态对比学习，并额外研究了同音素细化阶段。我们在Grad-TTS和潜在扩散语音合成系统中评估了内在文本-音频检索和下游合成。两阶段课程（MLM + 混合音素对比学习）在可懂度、说话人相似性和感知指标方面实现了最佳整体合成质量。尽管同音素细化提高了语调检索，但降低了音素辨别度并降低了合成质量。这些发现表明，嵌入空间指标的改进并不一定转化为更好的生成性能，强调了在语音合成预训练中平衡音素辨别度和语调敏感性的必要性。
英文摘要：
We investigate multi-stage pretraining for prosody modeling in diffusion-based TTS. A speaker-conditioned dual-stream encoder is trained with masked language modeling followed by SigLIP-style cross-modal contrastive learning using mixed-phoneme batches, with an additional same-phoneme refinement stage studied separately. We evaluate intrinsic text-audio retrieval and downstream synthesis in Grad-TTS and a latent diffusion TTS system. The two-stage curriculum (MLM + mixed-phoneme contrastive learning) achieves the best overall synthesis quality in terms of intelligibility, speaker similarity, and perceptual measures. Although same-phoneme refinement improves prosodic retrieval, it reduces phoneme discrimination and degrades synthesis. These findings indicate that improvements in embedding-space metrics do not necessarily translate to better generative performance and highlight the need to balance phoneme discrimination and prosodic sensitivity in TTS pretraining.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
