arXiv 2409.007502024-10-20cs.SDcs.AIcs.LGeess.AS

MaskGCT：基于掩码生成式编解码器Transformer的零样本文本到语音合成

MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer

The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Guangzhou Quwan Network Technology（广州曲湾网络科技）

Yuancheng Wang, Haoyue Zhan, Liwei Liu, Ruihong Zeng, Haotian Guo, Jiachen Zheng, Qiang Zhang, Xueyao Zhang, Shunsi Zhang, Zhizheng Wu

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出MaskGCT，一种完全非自回归的两阶段零样本TTS模型，通过掩码-预测范式并行生成语义和声学令牌，无需显式对齐和音素时长预测，在10万小时语音上超越现有最先进系统。

摘要：

近期的大规模文本到语音（TTS）系统通常分为自回归系统和非自回归系统。自回归系统隐式地建模时长，但在鲁棒性和时长可控性方面存在一定缺陷。非自回归系统在训练期间需要文本与语音之间的显式对齐信息，并为语言单元（如音素）预测时长，这可能损害其自然度。在本文中，我们提出了掩码生成式编解码器Transformer（MaskGCT），这是一种完全非自回归的TTS模型，消除了对文本与语音监督之间显式对齐信息以及音素级时长预测的需求。MaskGCT是一个两阶段模型：在第一阶段，模型使用文本预测从语音自监督学习（SSL）模型中提取的语义令牌；在第二阶段，模型以这些语义令牌为条件预测声学令牌。MaskGCT遵循掩码-预测学习范式。在训练期间，MaskGCT学习根据给定的条件和提示预测被掩码的语义或声学令牌。在推理期间，模型以并行方式生成指定长度的令牌。使用10万小时真实场景语音进行的实验表明，MaskGCT在质量、相似度和可懂度方面优于当前最先进的零样本TTS系统。音频样本可在https://maskgct.github.io/获取。我们在https://github.com/open-mmlab/Amphion/blob/main/models/tts/maskgct发布代码和模型检查点。

英文摘要：

The recent large-scale text-to-speech (TTS) systems are usually grouped as autoregressive and non-autoregressive systems. The autoregressive systems implicitly model duration but exhibit certain deficiencies in robustness and lack of duration controllability. Non-autoregressive systems require explicit alignment information between text and speech during training and predict durations for linguistic units (e.g. phone), which may compromise their naturalness. In this paper, we introduce Masked Generative Codec Transformer (MaskGCT), a fully non-autoregressive TTS model that eliminates the need for explicit alignment information between text and speech supervision, as well as phone-level duration prediction. MaskGCT is a two-stage model: in the first stage, the model uses text to predict semantic tokens extracted from a speech self-supervised learning (SSL) model, and in the second stage, the model predicts acoustic tokens conditioned on these semantic tokens. MaskGCT follows the mask-and-predict learning paradigm. During training, MaskGCT learns to predict masked semantic or acoustic tokens based on given conditions and prompts. During inference, the model generates tokens of a specified length in a parallel manner. Experiments with 100K hours of in-the-wild speech demonstrate that MaskGCT outperforms the current state-of-the-art zero-shot TTS systems in terms of quality, similarity, and intelligibility. Audio samples are available at https://maskgct.github.io/. We release our code and model checkpoints at https://github.com/open-mmlab/Amphion/blob/main/models/tts/maskgct.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。