arXiv 2409.121392024-09-24cs.SDcs.AIeess.AS

Takin：一组高质量零样本语音生成模型

Takin: A Cohort of Superior Quality Zero-shot Speech Generation Models

Cranberry-Lemon University（蔓越莓柠檬大学）

Sijing Chen, Yuan Feng, Laipeng He, Tianwei He, Wendi He, Yanni Hu, Bin Lin, Yiting Lin, Yu Pan, Pengfei Tan, Chengwei Tian, Chen Wang, Zhicheng Wang, Ruoye Xie… 展开作者

Sijing Chen, Yuan Feng, Laipeng He, Tianwei He, Wendi He, Yanni Hu, Bin Lin, Yiting Lin, Yu Pan, Pengfei Tan, Chengwei Tian, Chen Wang, Zhicheng Wang, Ruoye Xie, Jixun Yao, Quanlei Yan, Yuguang Yang, Jianhao Ye, Jingjing Yin, Yanzhen Yu, Huimin Zhang, Xiang Zhang, Guangcheng Zhao, Hongbin Zhou, Pengpeng Zou

备注：: Technical Report; 18 pages; typos corrected, references added, demo url modified, author name modified;

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文介绍专为有声书制作的Takin AudioLLM系列模型，通过Takin TTS、Takin VC和Takin Morphing实现高质量零样本语音生成与个性化定制，实验验证了其有效性与鲁棒性。

摘要：

随着大数据和大语言模型时代的到来，零样本个性化快速定制已成为一个重要趋势。在本报告中，我们介绍了Takin AudioLLM，这是一系列专为有声书制作设计的技术和模型，主要包括Takin TTS、Takin VC和Takin Morphing。这些模型能够进行零样本语音生产，生成与真实人类语音几乎无法区分的高质量语音，并使个人能够根据自身需求定制语音内容。具体而言，我们首先介绍Takin TTS，这是一种神经编解码器语言模型，基于增强型神经语音编解码器和多任务训练框架构建，能够以零样本方式生成高保真自然语音。对于Takin VC，我们提倡一种有效的内容与音色联合建模方法以提高说话人相似度，同时提倡基于条件流匹配的解码器以进一步增强其自然度和表现力。最后，我们提出了Takin Morphing系统，采用高度解耦且先进的音色和韵律建模方法，使个人能够以精确且可控的方式使用其偏好的音色和韵律定制语音生产。大量实验验证了Takin AudioLLM系列模型的有效性和鲁棒性。详细演示请参见 https://everest-ai.github.io/takinaudiollm/。

英文摘要：

With the advent of the big data and large language model era, zero-shot personalized rapid customization has emerged as a significant trend. In this report, we introduce Takin AudioLLM, a series of techniques and models, mainly including Takin TTS, Takin VC, and Takin Morphing, specifically designed for audiobook production. These models are capable of zero-shot speech production, generating high-quality speech that is nearly indistinguishable from real human speech and facilitating individuals to customize the speech content according to their own needs. Specifically, we first introduce Takin TTS, a neural codec language model that builds upon an enhanced neural speech codec and a multi-task training framework, capable of generating high-fidelity natural speech in a zero-shot way. For Takin VC, we advocate an effective content and timbre joint modeling approach to improve the speaker similarity, while advocating for a conditional flow matching based decoder to further enhance its naturalness and expressiveness. Last, we propose the Takin Morphing system with highly decoupled and advanced timbre and prosody modeling approaches, which enables individuals to customize speech production with their preferred timbre and prosody in a precise and controllable manner. Extensive experiments validate the effectiveness and robustness of our Takin AudioLLM series models. For detailed demos, please refer to https://everest-ai.github.io/takinaudiollm/.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。