arXiv 2407.054072024-07-09cs.SDcs.AIeess.AS

CosyVoice：一种基于有监督语义Token的可扩展多语言零样本文本转语音合成器

CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens

Alibaba Group（阿里巴巴集团）；Speech Lab（语音实验室）

Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, Zhifu Gao, Zhijie Yan

备注：: work in progress. arXiv admin note: substantial text overlap with arXiv:2407.04051

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对现有无监督语音Token缺乏语义、与文本不对齐的问题，提出基于有监督语义Token的多语言零样本TTS模型CosyVoice，在语音克隆效果和可扩展性上表现更优，为该领域首次尝试。

摘要：

近年来，基于大语言模型（LLM）的文本转语音（TTS）凭借高自然度和零样本能力成为主流趋势。在该范式下，语音信号被离散化为Token序列，由以文本为提示的LLM建模，再通过基于Token的声码器重建成波形。显然，语音Token在基于LLM的TTS模型中起着关键作用。当前语音Token以无监督方式学习，缺乏明确的语义信息且与文本不对齐。本文提出用有监督语义Token表示语音，这类Token通过在多语言语音识别模型的编码器中插入向量量化得到。基于这些Token，我们进一步提出可扩展的零样本TTS合成器CosyVoice，它由用于文本到Token生成的LLM和用于Token到语音合成的条件流匹配模型组成。实验结果表明，在零样本语音克隆的内容一致性和说话人相似度方面，有监督语义Token显著优于现有的无监督Token。此外，我们发现利用大规模数据可进一步提升合成性能，证明了CosyVoice的可扩展能力。据我们所知，这是首次将有监督语音Token引入TTS模型的尝试。

英文摘要：

Recent years have witnessed a trend that large language model (LLM) based text-to-speech (TTS) emerges into the mainstream due to their high naturalness and zero-shot capacity. In this paradigm, speech signals are discretized into token sequences, which are modeled by an LLM with text as prompts and reconstructed by a token-based vocoder to waveforms. Obviously, speech tokens play a critical role in LLM-based TTS models. Current speech tokens are learned in an unsupervised manner, which lacks explicit semantic information and alignment to the text. In this paper, we propose to represent speech with supervised semantic tokens, which are derived from a multilingual speech recognition model by inserting vector quantization into the encoder. Based on the tokens, we further propose a scalable zero-shot TTS synthesizer, CosyVoice, which consists of an LLM for text-to-token generation and a conditional flow matching model for token-to-speech synthesis. Experimental results show that supervised semantic tokens significantly outperform existing unsupervised tokens in terms of content consistency and speaker similarity for zero-shot voice cloning. Moreover, we find that utilizing large-scale data further improves the synthesis performance, indicating the scalable capacity of CosyVoice. To the best of our knowledge, this is the first attempt to involve supervised speech tokens into TTS models.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。