DUET: 扩散与流匹配驱动的文本转语音的统一双空间情感控制
DUET: Unified Dual-Space Emotion Control for Diffusion and Flow-Matching Driven Text-to-Speech
AI总结：
提出DUET框架，通过隐空间引导和梅尔谱梯度修正的双空间控制，在预训练扩散/流匹配TTS模型中实现细粒度情感控制，超越10个有监督基线。
摘要：
基于扩散和流匹配的文本转语音（TTS）模型在自然度方面表现出色，但由于情感信号与说话人身份纠缠，往往缺乏显式的情感控制。我们发现情感嵌入作为冻结隐藏状态的线性可解码方向出现，几乎与编码说话人身份的方向正交。这启发了一个即插即用框架DUET，用于对预训练的扩散和流匹配TTS模型进行情感控制。在生成过程中，DUET统一双空间控制，在单步更新中实现细粒度情感干预：隐空间引导沿目标情感方向移动生成，而梅尔谱引导通过从可微分声码器反向传播的梯度细化频谱细节。我们在三个数据集上的五个架构多样的预训练TTS骨干上验证了DUET，它跨范式超越了10个有监督的最先进情感TTS基线，并获得了最高的人类评分情感适宜性。为了进一步展示其定性行为，我们将DUET部署在Ameca人形机器人上，使其产生丰富表现力的情感语音，展示了即插即用情感交互在具身智能体中的巨大潜力。
英文摘要：
Diffusion and flow-matching based text-to-speech (TTS) models excel in naturalness but often lack explicit emotion control, as emotional signals remain entangled with speaker identity. We discover that emotion embedding emerges as a linearly decodable direction of frozen hidden states, nearly orthogonal to the direction embedding speaker identity. This inspires a plug-and-play framework DUET for emotion control over pretrained diffusion and flow-matching based TTS models. During generation, DUET unifies dual-space control to achieve fine-grained emotion intervention in a single per-step update: hidden space steering shifts generation along the target emotion direction, while mel-space guidance refines spectral details through gradients backpropagated from a differentiable vocoder. We validate DUET on five architecturally diverse pretrained TTS backbones across three datasets, where it outperforms 10 supervised state-of-the-art emotional TTS baselines across paradigms and achieves the highest human-rated emotion appropriateness. To further showcase its qualitative behavior, we deploy DUET on an Ameca humanoid robot, where it produces richly expressive emotional speech on the humanoid, demonstrating the strong potential for plug-and-play affective interaction for embodied agents.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
