跨语言F5-TTS：迈向语言无关的语音克隆与语音合成
Cross-Lingual F5-TTS: Towards Language-Agnostic Voice Cloning and Speech Synthesis
AI总结：
本文提出Cross-Lingual F5-TTS，通过强制对齐和说话速度预测实现无需转录文本的跨语言语音克隆与合成
摘要：
基于流匹配的文本到语音（TTS）模型已展现出高质量的语音合成。然而，大多数当前基于流匹配的TTS模型仍然依赖于与音频提示对应的参考转录文本进行合成。这种依赖性在音频提示转录文本不可用时阻碍了跨语言语音克隆，特别是对于未见过的语言。基于流匹配的TTS模型要消除音频提示转录文本的关键挑战在于训练期间识别词边界和推理期间确定适当的持续时间。在本文中，我们引入了Cross-Lingual F5-TTS框架，该框架能够在没有音频提示转录文本的情况下实现跨语言语音克隆。我们的方法通过强制对齐音频提示来预处理音频提示，以获得词边界，从而在训练期间排除转录文本的情况下直接从音频提示进行合成。为了解决持续时间建模的挑战，我们训练了不同语言颗粒度的说话速度预测器，以从说话速度推导出持续时间。实验表明，我们的方法在性能上与F5-TTS相匹配，同时实现了跨语言语音克隆。
英文摘要：
Flow-matching-based text-to-speech (TTS) models have shown high-quality speech synthesis. However, most current flow-matching-based TTS models still rely on reference transcripts corresponding to the audio prompt for synthesis. This dependency prevents cross-lingual voice cloning when audio prompt transcripts are unavailable, particularly for unseen languages. The key challenges for flow-matching-based TTS models to remove audio prompt transcripts are identifying word boundaries during training and determining appropriate duration during inference. In this paper, we introduce Cross-Lingual F5-TTS, a framework that enables cross-lingual voice cloning without audio prompt transcripts. Our method preprocesses audio prompts by forced alignment to obtain word boundaries, enabling direct synthesis from audio prompts while excluding transcripts during training. To address the duration modeling challenge, we train speaking rate predictors at different linguistic granularities to derive duration from speaker pace. Experiments show that our approach matches the performance of F5-TTS while enabling cross-lingual voice cloning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
