arXiv 2403.031002024-04-23eess.AScs.AIcs.CLcs.LGcs.SD

NaturalSpeech 3：基于分解编解码器与扩散模型的零样本语音合成

NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models

Microsoft Research（微软研究院）；Microsoft Azure（微软Azure）；University of Science and Technology of China（中国科学技术大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Zhejiang University（浙江大学）；The University of Tokyo（东京大学）；Peking University（北京大学）

Zeqian Ju, Yuancheng Wang, Kai Shen, Xu Tan, Detai Xin, Dongchao Yang, Yanqing Liu, Yichong Leng, Kaitao Song, Siliang Tang, Zhizheng Wu, Tao Qin, Xiang-Yang Li… 展开作者

Zeqian Ju, Yuancheng Wang, Kai Shen, Xu Tan, Detai Xin, Dongchao Yang, Yanqing Liu, Yichong Leng, Kaitao Song, Siliang Tang, Zhizheng Wu, Tao Qin, Xiang-Yang Li, Wei Ye, Shikun Zhang, Jiang Bian, Lei He, Jinyu Li, Sheng Zhao

备注：: Achieving human-level quality and naturalness on multi-speaker datasets (e.g., LibriSpeech) in a zero-shot way

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出NaturalSpeech 3系统，采用分解向量量化神经编解码器与分解扩散模型，将语音属性解耦至独立子空间并分别生成，在零样本语音合成任务中实现了优于现有系统的质量、相似度与可懂度，达到与人类录音相当的水平。

摘要：

尽管近期的大规模文本到语音（TTS）模型取得了显著进展，但它们在语音质量、相似度和韵律方面仍存在不足。考虑到语音错综复杂地包含各种属性（如内容、韵律、音色和声学细节），这给生成带来了巨大挑战，一个自然的想法是将语音分解为代表不同属性的独立子空间，并分别进行生成。受此启发，我们提出了 NaturalSpeech 3，这是一个采用新颖的分解扩散模型以零样本方式生成自然语音的 TTS 系统。具体而言，1）我们设计了一种带有分解向量量化（FVQ）的神经编解码器，将语音波形解耦为内容、韵律、音色和声学细节的子空间；2）我们提出了一种分解扩散模型，根据相应的提示在每个子空间中生成属性。通过这种分解设计，NaturalSpeech 3 能够以分而治之的方式，有效且高效地对具有解耦子空间的复杂语音进行建模。实验表明，NaturalSpeech 3 在质量、相似度、韵律和可懂度上优于最先进的 TTS 系统，并实现了与人类录音相当的质量。此外，通过扩展至 1B 参数和 200K 小时的训练数据，我们取得了更好的性能。

英文摘要：

While recent large-scale text-to-speech (TTS) models have achieved significant progress, they still fall short in speech quality, similarity, and prosody. Considering speech intricately encompasses various attributes (e.g., content, prosody, timbre, and acoustic details) that pose significant challenges for generation, a natural idea is to factorize speech into individual subspaces representing different attributes and generate them individually. Motivated by it, we propose NaturalSpeech 3, a TTS system with novel factorized diffusion models to generate natural speech in a zero-shot way. Specifically, 1) we design a neural codec with factorized vector quantization (FVQ) to disentangle speech waveform into subspaces of content, prosody, timbre, and acoustic details; 2) we propose a factorized diffusion model to generate attributes in each subspace following its corresponding prompt. With this factorization design, NaturalSpeech 3 can effectively and efficiently model intricate speech with disentangled subspaces in a divide-and-conquer way. Experiments show that NaturalSpeech 3 outperforms the state-of-the-art TTS systems on quality, similarity, prosody, and intelligibility, and achieves on-par quality with human recordings. Furthermore, we achieve better performance by scaling to 1B parameters and 200K hours of training data.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。