VisualTTS:TTS,具有精确的唇语同步功能,用于自动配音
VisualTTS: TTS with Accurate Lip-Speech Synchronization for Automatic Voice Over
摘要:
在本文中,我们制定了一个新的任务,合成语音同步与无声的预先录制的视频,表示为自动配音(AVO)。与传统的语音合成不同,AVO不仅要生成听起来像人的语音,而且要实现完美的唇语同步。AVO的一个自然解决方案是将语音渲染的条件设定在视频中嘴唇序列的时间进展上。我们提出了一种新的文本到语音的模型,条件是视觉输入,命名为VisualTTS,准确的唇语同步。该算法采用了两种新颖的机制:1)文本-视觉注意机制和2)声学解码过程中的视觉融合策略,这两种机制都有助于在输入文本内容和输入嘴唇序列中的嘴唇运动之间形成准确的对齐。实验结果表明,VisualTTS实现了准确的唇语同步,并优于所有基线系统。
英文摘要:
In this paper, we formulate a novel task to synthesize speech in sync with a silent pre-recorded video, denoted as automatic voice over (AVO). Unlike traditional speech synthesis, AVO seeks to generate not only human-sounding speech, but also perfect lip-speech synchronization. A natural solution to AVO is to condition the speech rendering on the temporal progression of lip sequence in the video. We propose a novel text-to-speech model that is conditioned on visual input, named VisualTTS, for accurate lip-speech synchronization. The proposed VisualTTS adopts two novel mechanisms that are 1) textual-visual attention, and 2) visual fusion strategy during acoustic decoding, which both contribute to forming accurate alignment between the input text content and lip motion in input lip sequence. Experimental results show that VisualTTS achieves accurate lip-speech synchronization and outperforms all baseline systems.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。