GOAT-TTS：基于双分支LLM的表达性且真实的语音生成
GOAT-TTS: Expressive and Realistic Speech Generation via A Dual-Branch LLM
AI总结：
针对当前TTS架构在声学特征丢失、严格对齐依赖和文本理解灾难性遗忘上的矛盾，提出双分支架构GOAT-TTS，通过模态对齐与模块化微调实现无需转录依赖的实时流式高质量语音合成。
摘要：
尽管大型语言模型（LLM）通过离散令牌化范式彻底改变了文本转语音（TTS）合成，但当前架构在三个关键维度上表现出根本性矛盾：1）由语音提示量化导致的声学特征不可逆丢失；2）对精确对齐的提示语音-文本对的严格依赖，限制了现实部署；3）在优化语音令牌生成时，LLM原生文本理解能力发生灾难性遗忘。为应对这些挑战，我们提出了一种基于LLM的文本转语音生成方法，通过新颖的双分支架构进行优化（GOAT-TTS）。我们的框架引入了两项关键创新：（1）模态对齐分支结合语音编码器和投影器，捕获连续声学嵌入，在无转录依赖的情况下实现副语言特征（语言、音色、情感）与语义文本表示之间的双向关联；（2）语音生成分支在LLM的top-k层进行模块化微调以预测语音令牌，同时冻结bottom-n层以保留基础语言学知识。此外，引入多令牌预测以支持实时流式TTS合成。实验结果表明，我们的GOAT-TTS取得了与最先进TTS模型相当的性能，同时验证了合成方言语音数据的有效性。
英文摘要：
While large language models (LLMs) have revolutionized text-to-speech (TTS) synthesis through discrete tokenization paradigms, current architectures exhibit fundamental tensions between three critical dimensions: 1) irreversible loss of acoustic characteristics caused by quantization of speech prompts; 2) stringent dependence on precisely aligned prompt speech-text pairs that limit real-world deployment; and 3) catastrophic forgetting of the LLM's native text comprehension during optimization for speech token generation. To address these challenges, we propose an LLM-based text-to-speech Generation approach Optimized via a novel dual-branch ArchiTecture (GOAT-TTS). Our framework introduces two key innovations: (1) The modality-alignment branch combines a speech encoder and projector to capture continuous acoustic embeddings, enabling bidirectional correlation between paralinguistic features (language, timbre, emotion) and semantic text representations without transcript dependency; (2) The speech-generation branch employs modular fine-tuning on top-k layers of an LLM for speech token prediction while freezing the bottom-n layers to preserve foundational linguistic knowledge. Moreover, multi-token prediction is introduced to support real-time streaming TTS synthesis. Experimental results demonstrate that our GOAT-TTS achieves performance comparable to state-of-the-art TTS models while validating the efficacy of synthesized dialect speech data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
