OpenVoice：多功能即时语音克隆
OpenVoice: Versatile Instant Voice Cloning
AI总结：
本研究提出OpenVoice多功能即时语音克隆方法，实现灵活语音风格控制与零样本跨语言克隆，计算成本远低于商用API，已开源并获大规模应用。
摘要：
我们提出了OpenVoice，这是一种多功能语音克隆方法，仅需参考说话人的一段短音频片段，就能复刻其嗓音并生成多语言语音。OpenVoice在解决该领域以下未决挑战方面取得了重大进展：1）灵活的语音风格控制。除了复刻参考说话人的音色外，OpenVoice还能对语音风格进行细粒度控制，包括情感、口音、节奏、停顿和语调。这些语音风格并非直接从参考说话人的风格中复制而来，也不受其限制。此前的方法不具备克隆后灵活调整语音风格的能力。2）零样本跨语言语音克隆。OpenVoice针对大规模说话人训练集中未包含的语言，实现了零样本跨语言语音克隆。此前的方法通常需要为所有语言准备大规模说话人多语言（MSML）数据集，而OpenVoice无需目标语言的任何大规模说话人训练数据，就能将语音克隆到该新语言中。OpenVoice还具备计算高效的特点，其成本比性能更差的商用API低数十倍。为推动该领域的进一步研究，我们已将源代码和训练好的模型公开。我们还在演示网站上提供了定性结果。作为MyShell.ai的语音引擎，OpenVoice已在全球拥有超过200万用户。
英文摘要：
We introduce OpenVoice, a versatile voice cloning approach that requires only a short audio clip from the reference speaker to replicate their voice and generate speech in multiple languages. OpenVoice represents a significant advancement in addressing the following open challenges in the field: 1) Flexible Voice Style Control. OpenVoice enables granular control over voice styles, including emotion, accent, rhythm, pauses, and intonation, in addition to replicating the tone color of the reference speaker. The voice styles are not directly copied from and constrained by the style of the reference speaker. Previous approaches lacked the ability to flexibly manipulate voice styles after cloning. 2) Zero-Shot Cross-Lingual Voice Cloning. OpenVoice achieves zero-shot cross-lingual voice cloning for languages not included in the massive-speaker training set. Unlike previous approaches, which typically require extensive massive-speaker multi-lingual (MSML) dataset for all languages, OpenVoice can clone voices into a new language without any massive-speaker training data for that language. OpenVoice is also computationally efficient, costing tens of times less than commercially available APIs that offer even inferior performance. To foster further research in the field, we have made the source code and trained model publicly accessible. We also provide qualitative results in our demo website. OpenVoice has been used by more than 2M users worldwide as the voice engine of MyShell.ai
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
