JoyVoice: 长上下文条件生成用于拟人化多说话人对话合成
JoyVoice: Long-Context Conditioning for Anthropomorphic Multi-Speaker Conversational Synthesis
AI总结：
JoyVoice通过统一的E2E-Transformer-DiT架构实现多说话人长对话合成，取得多语言生成和零样本语音克隆的先进成果。
摘要：
大型语音生成模型正从单说话人、短句合成发展到多说话人、长对话生成。当前长形式语音生成模型主要受限于二元、回合制交互。为解决这一问题，我们引入JoyVoice，一种新的拟人化基础模型，用于灵活、无边界的最多八说话人合成。与传统级联系统不同，JoyVoice采用统一的E2E-Transformer-DiT架构，利用自回归隐藏表示直接用于扩散输入，实现整体端到端优化。我们进一步提出一个在低比特率（12.5 Hz）下运行的MM-Tokenizer，整合多任务语义和MMSE损失，以有效建模语义和语音信息。此外，模型还通过大规模数据扰动实现稳健的文本前端处理。实验表明，JoyVoice在多语言生成（中文、英文、日文、韩文）和零样本语音克隆中实现了最先进的结果。JoyVoice在Seed-TTS-Eval基准和多说话人长形式对话语音克隆任务中均取得顶级结果，展示出优越的音频质量和泛化能力。它在长形式语音的语调连续性、多说话人对话的节奏丰富性、非语言自然性和优越的可懂性方面实现了显著改进。我们鼓励读者在https://jea-speech.github.io/JoyVoice上聆听演示。
英文摘要：
Large speech generation models are evolving from single-speaker, short sentence synthesis to multi-speaker, long conversation geneartion. Current long-form speech generation models are predominately constrained to dyadic, turn-based interactions. To address this, we introduce JoyVoice, a novel anthropomorphic foundation model designed for flexible, boundary-free synthesis of up to eight speakers. Unlike conventional cascaded systems, JoyVoice employs a unified E2E-Transformer-DiT architecture that utilizes autoregressive hidden representations directly for diffusion inputs, enabling holistic end-to-end optimization. We further propose a MM-Tokenizer operating at a low bitrate of 12.5 Hz, which integrates multitask semantic and MMSE losses to effectively model both semantic and acoustic information. Additionally, the model incorporates robust text front-end processing via large-scale data perturbation. Experiments show that JoyVoice achieves state-of-the-art results in multilingual generation (Chinese, English, Japanese, Korean) and zero-shot voice cloning. JoyVoice achieves top-tier results on both the Seed-TTS-Eval Benchmark and multi-speaker long-form conversational voice cloning tasks, demonstrating superior audio quality and generalization. It achieves significant improvements in prosodic continuity for long-form speech, rhythm richness in multi-speaker conversations, paralinguistic naturalness, besides superior intelligibility. We encourage readers to listen to the demo at https://jea-speech.github.io/JoyVoice
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
