理解与表达：通过双语言建模实现文本转语音合成
Comprehend and Talk: Text to Speech Synthesis via Dual Language Modeling
AI总结：
本文提出CaT-TTS框架，结合语义增强的S3Codec、理解-生成双Transformer架构与MAPI推理策略，以提升零样本TTS的语义建模能力与生成稳定性。
摘要：
现有的基于大语言模型（LLM）的自回归（AR）文本转语音（TTS）系统虽然达到了最先进的质量，但仍面临关键挑战。这种基于LLM范式的基础，是通过神经音频编解码器将连续语音波形离散化为离散token序列。然而，单码本建模非常适合文本LLM，却会遭受显著的信息损失；通常通过残差矢量量化（RVQ）生成的分层声学token往往缺乏显式语义结构，给模型带来沉重的学习负担。此外，自回归过程本身容易受到误差累积的影响，从而降低生成稳定性。为解决这些局限，我们提出CaT-TTS，这是一种用于鲁棒且基于语义的零样本合成的新颖框架。首先，我们提出S3Codec，这是一种拆分式RVQ编解码器，通过来自最先进ASR模型的语义蒸馏，将显式语言特征注入其主码本，从而提供可简化学习任务的结构化表示。其次，我们提出“先理解后生成”的双Transformer架构，将理解与渲染解耦。初始的“理解”Transformer对文本与音频语义token之间的跨模态关系进行建模，以形成高层话语计划。随后的“生成”Transformer执行该计划，自回归地合成分层声学token。最后，为增强生成稳定性，我们引入掩码音频并行推理（MAPI），这是一种几乎无参数的推理策略，可动态引导解码过程以缓解局部误差。
英文摘要：
Existing Large Language Model (LLM) based autoregressive (AR) text-to-speech (TTS) systems, while achieving state-of-the-art quality, still face critical challenges. The foundation of this LLM-based paradigm is the discretization of the continuous speech waveform into a sequence of discrete tokens by neural audio codec. However, single codebook modeling is well suited to text LLMs, but suffers from significant information loss; hierarchical acoustic tokens, typically generated via Residual Vector Quantization (RVQ), often lack explicit semantic structure, placing a heavy learning burden on the model. Furthermore, the autoregressive process is inherently susceptible to error accumulation, which can degrade generation stability. To address these limitations, we propose CaT-TTS, a novel framework for robust and semantically-grounded zero-shot synthesis. First, we introduce S3Codec, a split RVQ codec that injects explicit linguistic features into its primary codebook via semantic distillation from a state-of-the-art ASR model, providing a structured representation that simplifies the learning task. Second, we propose an ``Understand-then-Generate'' dual-Transformer architecture that decouples comprehension from rendering. An initial ``Understanding'' Transformer models the cross-modal relationship between text and the audio's semantic tokens to form a high-level utterance plan. A subsequent ``Generation'' Transformer then executes this plan, autoregressively synthesizing hierarchical acoustic tokens. Finally, to enhance generation stability, we introduce Masked Audio Parallel Inference (MAPI), a nearly parameter-free inference strategy that dynamically guides the decoding process to mitigate local errors.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
