arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.12229eess.AScs.AIeess.SP
收藏

Laugh Now Cry Later:控制基于流匹配的零样本文本转语音的时变情绪状态

Laugh Now Cry Later: Controlling Time-Varying Emotional States of Flow-Matching-Based Zero-Shot Text-to-Speech

Microsoft(微软公司)

Haibin Wu, Xiaofei Wang, Sefik Emre Eskimez, Manthan Thakker, Daniel Tompkins, Chung-Hsien Tsai, Canrun Li, Zhen Xiao, Sheng Zhao, Jinyu Li, Naoyuki Kanda

AI总结:

针对现有TTS系统难以生成含非语言发声的丰富情绪语音的问题,提出情绪可控零样本模型EmoCtrl-TTS,通过情绪维度与笑声嵌入约束流匹配模型,经27000小时数据训练,在情绪模仿、情绪变化捕捉及非语言发声生成上表现优异。

摘要:

人们会改变语调,常伴随笑声、哭声等非语言发声(NVs),以传递丰富情绪。但大多数文本转语音(TTS)系统无法生成包含非语言发声在内的富有情绪的语音。本文提出EmoCtrl-TTS,这是一种情绪可控的零样本TTS,可为任意说话人生成带有非语言发声的高情绪强度语音。EmoCtrl-TTS利用唤醒度、效价值以及笑声嵌入,对基于流匹配的零样本TTS进行条件约束。为实现高质量情绪语音生成,EmoCtrl-TTS使用超过27000小时经伪标注筛选的富有表现力的数据训练。综合评估表明,在语音转语音翻译场景中,EmoCtrl-TTS在模仿音频提示的情绪方面表现优异。我们还展示了EmoCtrl-TTS可捕捉情绪变化、表达强烈情绪,并能在零样本TTS中生成多种非语言发声。演示样本见https://aka.ms/emoctrl-tts。

英文摘要:

People change their tones of voice, often accompanied by nonverbal vocalizations (NVs) such as laughter and cries, to convey rich emotions. However, most text-to-speech (TTS) systems lack the capability to generate speech with rich emotions, including NVs. This paper introduces EmoCtrl-TTS, an emotion-controllable zero-shot TTS that can generate highly emotional speech with NVs for any speaker. EmoCtrl-TTS leverages arousal and valence values, as well as laughter embeddings, to condition the flow-matching-based zero-shot TTS. To achieve high-quality emotional speech generation, EmoCtrl-TTS is trained using more than 27,000 hours of expressive data curated based on pseudo-labeling. Comprehensive evaluations demonstrate that EmoCtrl-TTS excels in mimicking the emotions of audio prompts in speech-to-speech translation scenarios. We also show that EmoCtrl-TTS can capture emotion changes, express strong emotions, and generate various NVs in zero-shot TTS. See https://aka.ms/emoctrl-tts for demo samples.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑