CapTalk：单语句和对话语音生成的统一语音设计
CapTalk: Unified Voice Design for Single-Utterance and Dialogue Speech Generation
AI总结：
CapTalk提出一种统一的文本-音频自回归框架，实现单语句和对话语音设计，通过分层变分条件模块平衡语音稳定性和上下文适应性，提升表达可控性和上下文恰当性。
摘要：
从自然语言描述中生成语音的设计正在成为文本到语音多模态生成中的新任务，旨在在不依赖参考音频的情况下合成具有目标音色和说话风格的语音。然而，现有方法主要集中在单语句生成，对话语音设计仍待探索。本文将语音设计扩展到对话场景，实现更好的目标说话人建模和回合级表达控制。我们提出CapTalk，一种统一的caption-conditioned文本-音频自回归框架，用于单语句和对话语音设计。CapTalk使用语句级caption进行单语句语音设计，使用说话人级caption进行对话说话人建模，并进一步引入CoT控制序列在对话中显式规划回合级动态属性。为解决稳定音色保留与上下文适应表达之间的冲突，我们提出一个分层变分条件模块，包含语句级说话人编码器，以更好地平衡稳定音色保留和上下文适应表达。这使得在保持当前语句表达适应的同时实现音色重用，并在对话中保持上下文适应性。我们还构建了单语句和对话设置的综合评估协议。实验表明，CapTalk在单语句语音设计基准上实现了最先进的性能，并在多轮对话中实现了更好的表达可控性和上下文恰当性。音频样本可在：https://anonymous.4open.science/api/repo/Captalk-D601/file/index.html获取。
英文摘要：
Voice design from natural language descriptions is emerging as a new task in text-to-speech multimodal generation, aiming to synthesize speech with target timbre and speaking style without relying on reference audio. However, existing methods mainly focus on single-utterance generation, leaving conversational voice design largely unexplored. In this work, we extend voice design to dialogue, enabling better target speaker modeling and turn-level expressive control in natural conversational settings. We propose CapTalk, a unified caption-conditioned text-audio autoregressive framework for both single-utterance and dialogue voice design. CapTalk uses utterance-level captions for single-utterance voice design and speaker-level captions for dialogue speaker modeling, and further introduces a CoT control sequence in dialogue to explicitly plan turn-level dynamic attributes. To resolve the conflict between stable timbre preservation and context-adaptive expression, we propose a hierarchical variational conditioning module with an utterance-level speaker encoder to better balance stable timbre preservation and context-adaptive expression. This enables timbre reuse while keeping expression adaptive to the current utterance and, in dialogue, the surrounding context. We also build a comprehensive evaluation protocol for both single-utterance and dialogue settings. Experiments show that CapTalk achieves state-of-the-art performance on a single-utterance voice design benchmark and delivers better expression controllability and contextual appropriateness in multi-turn dialogue. Audio samples are available at: https://anonymous.4open.science/api/repo/Captalk-D601/file/index.html.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
