LatinX：通过直接偏好优化对齐多语言文本到语音模型
LatinX: Aligning a Multilingual TTS Model with Direct Preference Optimization
AI总结：
本文提出LatinX，一种三阶段训练的多语言TTS模型，通过DPO对齐降低WER并提升说话人相似度，在跨语言语音翻译中优于基线XTTSv2。
摘要：
我们提出了LatinX，一种用于级联语音到语音翻译的多语言文本到语音（TTS）模型，该模型能够跨语言保留源说话人的身份。LatinX是一个12层仅解码器的Transformer，分三个阶段进行训练：（i）文本到音频映射的预训练，（ii）用于零样本语音克隆的有监督微调，以及（iii）使用基于词错误率（WER）和说话人相似度指标自动标注的偏好对，通过直接偏好优化（DPO）进行对齐。LatinX在英语和罗曼语系语言（重点是葡萄牙语）上训练，与微调基线相比，使用DPO的LatinX持续降低WER并提高客观相似度。人工评估进一步表明，与强基线（XTTSv2）相比，感知到的说话人相似度更强，揭示了客观与主观度量之间的差距。我们提供了跨语言分析，并讨论了平衡的偏好信号和低延迟架构作为未来工作。
英文摘要：
We present LatinX, a multilingual text-to-speech (TTS) model for cascaded speech-to-speech translation that preserves the source speaker's identity across languages. LatinX is a 12-layer decoder-only Transformer trained in three stages: (i) pre-training for text-to-audio mapping, (ii) supervised fine-tuning for zero-shot voice cloning, and (iii) alignment with Direct Preference Optimization (DPO) using automatically labeled pairs based on Word Error Rate (WER) and speaker-similarity metrics. Trained on English and Romance languages with emphasis on Portuguese, LatinX with DPO consistently reduces WER and improves objective similarity over the fine-tuned baseline. Human evaluations further indicate stronger perceived speaker similarity than a strong baseline (XTTSv2), revealing gaps between objective and subjective measures. We provide cross-lingual analyses and discuss balanced preference signals and lower-latency architectures as future work.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
