arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2110.03342eess.AScs.CL
收藏

VisualTTS:TTS,具有精确的唇语同步功能,用于自动配音

VisualTTS: TTS with Accurate Lip-Speech Synchronization for Automatic Voice Over

National University of Singapore(新加坡国立大学);Singapore University of Technology and Design(新加坡科技设计大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳))

Junchen Lu, Berrak Sisman, Rui Liu, Mingyang Zhang, Haizhou Li

摘要:

在本文中,我们制定了一个新的任务,合成语音同步与无声的预先录制的视频,表示为自动配音(AVO)。与传统的语音合成不同,AVO不仅要生成听起来像人的语音,而且要实现完美的唇语同步。AVO的一个自然解决方案是将语音渲染的条件设定在视频中嘴唇序列的时间进展上。我们提出了一种新的文本到语音的模型,条件是视觉输入,命名为VisualTTS,准确的唇语同步。该算法采用了两种新颖的机制:1)文本-视觉注意机制和2)声学解码过程中的视觉融合策略,这两种机制都有助于在输入文本内容和输入嘴唇序列中的嘴唇运动之间形成准确的对齐。实验结果表明,VisualTTS实现了准确的唇语同步,并优于所有基线系统。

英文摘要:

In this paper, we formulate a novel task to synthesize speech in sync with a silent pre-recorded video, denoted as automatic voice over (AVO). Unlike traditional speech synthesis, AVO seeks to generate not only human-sounding speech, but also perfect lip-speech synchronization. A natural solution to AVO is to condition the speech rendering on the temporal progression of lip sequence in the video. We propose a novel text-to-speech model that is conditioned on visual input, named VisualTTS, for accurate lip-speech synchronization. The proposed VisualTTS adopts two novel mechanisms that are 1) textual-visual attention, and 2) visual fusion strategy during acoustic decoding, which both contribute to forming accurate alignment between the input text content and lip motion in input lip sequence. Experimental results show that VisualTTS achieves accurate lip-speech synchronization and outperforms all baseline systems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑