Fish-Speech：利用大型语言模型实现先进的多语言文本到语音合成
Fish-Speech: Leveraging Large Language Models for Advanced Multilingual Text-to-Speech Synthesis
AI总结：
Fish-Speech通过双自回归架构和大型语言模型，无需G2P转换，实现高保真多语言语音合成，在复杂场景和语音克隆上显著优于基线。
摘要：
文本到语音（TTS）系统在处理复杂语言特征、多义表达以及生成自然的多语言语音方面面临持续挑战，而这些能力对于未来的人工智能应用至关重要。在本文中，我们提出了Fish-Speech，一个新颖的框架，它实现了串行快慢双自回归（Dual-AR）架构，以增强分组有限标量矢量量化（GFSQ）在序列生成任务中的稳定性。该架构提高了码本处理效率，同时保持高保真输出，使其特别适用于人工智能交互和语音克隆。Fish-Speech利用大型语言模型（LLMs）进行语言特征提取，消除了传统字形到音素（G2P）转换的需求，从而简化了合成流程并增强了多语言支持。此外，我们通过GFSQ开发了FF-GAN，以实现优越的压缩比和接近100%的码本利用率。我们的方法解决了当前TTS系统的关键局限性，同时为更复杂、上下文感知的语音合成提供了基础。实验结果表明，Fish-Speech在处理复杂语言场景和语音克隆任务方面显著优于基线模型，展示了其在人工智能应用中推进TTS技术的潜力。该实现已在https://github.com/fishaudio/fish-speech开源。
英文摘要：
Text-to-Speech (TTS) systems face ongoing challenges in processing complex linguistic features, handling polyphonic expressions, and producing natural-sounding multilingual speech - capabilities that are crucial for future AI applications. In this paper, we present Fish-Speech, a novel framework that implements a serial fast-slow Dual Autoregressive (Dual-AR) architecture to enhance the stability of Grouped Finite Scalar Vector Quantization (GFSQ) in sequence generation tasks. This architecture improves codebook processing efficiency while maintaining high-fidelity outputs, making it particularly effective for AI interactions and voice cloning. Fish-Speech leverages Large Language Models (LLMs) for linguistic feature extraction, eliminating the need for traditional grapheme-to-phoneme (G2P) conversion and thereby streamlining the synthesis pipeline and enhancing multilingual support. Additionally, we developed FF-GAN through GFSQ to achieve superior compression ratios and near 100\% codebook utilization. Our approach addresses key limitations of current TTS systems while providing a foundation for more sophisticated, context-aware speech synthesis. Experimental results show that Fish-Speech significantly outperforms baseline models in handling complex linguistic scenarios and voice cloning tasks, demonstrating its potential to advance TTS technology in AI applications. The implementation is open source at \href{https://github.com/fishaudio/fish-speech}{https://github.com/fishaudio/fish-speech}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
