VoxCPM：用于上下文感知语音生成和逼真声音克隆的无分词器 TTS
VoxCPM: Tokenizer-Free TTS for Context-Aware Speech Generation and True-to-Life Voice Cloning
AI总结：
本文提出无分词器 TTS 模型 VoxCPM，通过半离散残差表示、TSLM/RALM 分层建模与端到端扩散解码器，在 180 万小时双语语料上训练并实现开源最优零样本语音合成。
摘要：
用于语音合成的生成模型面临一个根本性权衡：离散 token 确保稳定性，但会牺牲表现力；连续信号保留声学丰富性，却因任务纠缠而遭受误差累积。这一挑战促使该领域转向依赖预训练语音分词器的多阶段流水线，但这些方法造成了语义—声学鸿沟，限制了整体性、富有表现力的语音生成。我们通过采用半离散残差表示的分层语义—声学建模来解决这一困境，并提出一种新颖的无分词器 TTS 模型 VoxCPM。我们的框架引入可微量化瓶颈，从而产生自然的专业化分工：Text-Semantic Language Model（TSLM，文本—语义语言模型）生成语义—韵律计划，而 Residual Acoustic Model（RALM，残差声学模型）恢复细粒度声学细节。这种分层语义—声学表示引导基于局部扩散的解码器生成高保真语音潜变量。关键在于，整个架构在简单的扩散目标下进行端到端训练，消除了对外部语音分词器的依赖。我们的 VoxCPM-0.5B 模型在 180 万小时的大规模双语语料上训练，在开源系统中取得了最先进的零样本 TTS 性能，表明我们的方法可实现富有表现力且稳定的合成。此外，VoxCPM 展现出理解文本以推断并生成适当韵律和风格的能力，可交付具有上下文感知表现力和自然流畅度的语音。为促进社区驱动的研发，VoxCPM 依据 Apache 2.0 公开开放。
英文摘要：
Generative models for speech synthesis face a fundamental trade-off: discrete tokens ensure stability but sacrifice expressivity, while continuous signals retain acoustic richness but suffer from error accumulation due to task entanglement. This challenge has driven the field towards multi-stage pipelines that rely on pre-trained speech tokenizers, but these create a semantic-acoustic divide, limiting holistic and expressive speech generation. We resolve these dilemma through hierarchical semantic-acoustic modeling with semi-discrete residual representations and present a novel tokenizer-free TTS model VoxCPM. Our framework introduces a differentiable quantization bottleneck that induces natural specialization: a Text-Semantic Language Model (TSLM) generates semantic-prosodic plans, while a Residual Acoustic Model (RALM) recovers fine-grained acoustic details. This hierarchical semantic-acoustic representation guides a local diffusion-based decoder to generate high-fidelity speech latents. Critically, the entire architecture is trained end-to-end under a simple diffusion objective, eliminating dependency on external speech tokenizers. Trained on a massive 1.8 million hours of bilingual corpus, our VoxCPM-0.5B model achieves state-of-the-art zero-shot TTS performance among open-source systems, demonstrating that our approach delivers expressive and stable synthesis. Besides, VoxCPM shows the capability to comprehend text to infer and generate appropriate prosody and style, delivering speech with context-aware expressiveness and natural flow. To facilitate community-driven research and development, VoxCPM is publicly accessible under Apache 2.0.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
