UniTTS：一种不分离声学与语义信息的端到端TTS系统
UniTTS: An end-to-end TTS system without decoupling of acoustic and semantic information
AI总结：
针对LLM-TTS中语义与声学信息不对齐导致音频信息受限的问题，提出DistilCodec单码本编解码器及UniTTS端到端系统，通过三阶段训练整合跨模态自回归，实现高效语音合成。
摘要：
多码本中性音频编解码器（如残差向量量化（RVQ）和组向量量化（GVQ））的出现，显著推进了基于大语言模型（LLM）的文本到语音（TTS）系统。这些编解码器在分离语义和声学信息的同时，有效利用语义先验方面至关重要。然而，由于语义和声学信息无法完全对齐，这些方法应用于基于LLM的TTS时的一个显著缺点是，大语言模型可能难以获取全面的音频信息。为解决这一限制，我们提出了DistilCodec和UniTTS，它们共同具有以下优势：1）该方法可将多码本音频编解码器蒸馏为具有32,768个码的单码本音频编解码器，同时实现接近100%的利用率。2）由于DistilCodec不采用语义对齐方案，训练过程中可纳入大量高质量的无标注音频（如带有音效的有声书、歌曲等），进一步扩展数据多样性并拓宽其适用性。3）利用DistilCodec全面的音频信息建模能力，我们将三个关键任务整合到UniTTS的预训练框架中：音频模态自回归、文本模态自回归以及语音-文本跨模态自回归。这使得UniTTS能够接受交错的文本和语音/音频提示，同时大幅保留LLM的文本能力。4）UniTTS采用三阶段训练流程：预训练、监督微调（SFT）和对齐。源代码和模型检查点已在https://github.com/IDEA-Emdoor-Lab/UniTTS和https://github.com/IDEA-Emdoor-Lab/DistilCodec公开提供。
英文摘要：
The emergence of multi-codebook neutral audio codecs such as Residual Vector Quantization (RVQ) and Group Vector Quantization (GVQ) has significantly advanced Large-Language-Model (LLM) based Text-to-Speech (TTS) systems. These codecs are crucial in separating semantic and acoustic information while efficiently harnessing semantic priors. However, since semantic and acoustic information cannot be fully aligned, a significant drawback of these methods when applied to LLM-based TTS is that large language models may have limited access to comprehensive audio information. To address this limitation, we propose DistilCodec and UniTTS, which collectively offer the following advantages: 1) This method can distill a multi-codebook audio codec into a single-codebook audio codec with 32,768 codes while achieving a near 100\% utilization. 2) As DistilCodec does not employ a semantic alignment scheme, a large amount of high-quality unlabeled audio (such as audiobooks with sound effects, songs, etc.) can be incorporated during training, further expanding data diversity and broadening its applicability. 3) Leveraging the comprehensive audio information modeling of DistilCodec, we integrated three key tasks into UniTTS's pre-training framework: audio modality autoregression, text modality autoregression, and speech-text cross-modal autoregression. This allows UniTTS to accept interleaved text and speech/audio prompts while substantially preserving LLM's text capabilities. 4) UniTTS employs a three-stage training process: Pre-Training, Supervised Fine-Tuning (SFT), and Alignment. Source code and model checkpoints are publicly available at https://github.com/IDEA-Emdoor-Lab/UniTTS and https://github.com/IDEA-Emdoor-Lab/DistilCodec.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
