arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2106.04283cs.SDcs.AIcs.CVcs.LGeess.AS
收藏

NWT:通过表示学习实现自然的音频到视频生成

NWT: Towards natural audio-to-video generation with representation learning

Cash App Labs(Cash App实验室)

Rayhane Mama, Marc S. Tyndel, Hashiam Kadhim, Cole Clifford, Ragavan Thurairatnam

摘要:

在这项工作中,我们介绍了NWT,一个富有表现力的语音到视频模型。与使用特定领域的中间表示(如姿势关键点)的方法不同,NWT学习自己的潜在表示,对音频和视频内容的假设最少。为此,我们提出了一种新的离散变分自编码器与对抗损失,dVAE-Adv,学习一个新的离散的潜在表示,我们称之为Memcodes。Memcodes是简单的实现,不需要额外的损失条款,是稳定的训练与其他方法相比,并显示出可解释性的证据。为了在Memcode空间上进行预测,我们使用以音频为条件的自回归编码器-解码器模型。此外,我们的模型可以控制生成的视频中未在数据中注释的潜在属性。我们训练NWT的剪辑从HBO的最后一周今晚与约翰奥利弗。NWT在整体视频自然度、面部自然度和表现力以及对口型质量测试中的平均意见得分(MOS)始终高于其他方法。这项工作为广义的音频到视频合成设置了一个强有力的基线。样品可在https://next-week-tonight.github.io/NWT/上获得。

英文摘要:

In this work we introduce NWT, an expressive speech-to-video model. Unlike approaches that use domain-specific intermediate representations such as pose keypoints, NWT learns its own latent representations, with minimal assumptions about the audio and video content. To this end, we propose a novel discrete variational autoencoder with adversarial loss, dVAE-Adv, which learns a new discrete latent representation we call Memcodes. Memcodes are straightforward to implement, require no additional loss terms, are stable to train compared with other approaches, and show evidence of interpretability. To predict on the Memcode space, we use an autoregressive encoder-decoder model conditioned on audio. Additionally, our model can control latent attributes in the generated video that are not annotated in the data. We train NWT on clips from HBO's Last Week Tonight with John Oliver. NWT consistently scores above other approaches in Mean Opinion Score (MOS) on tests of overall video naturalness, facial naturalness and expressiveness, and lipsync quality. This work sets a strong baseline for generalized audio-to-video synthesis. Samples are available at https://next-week-tonight.github.io/NWT/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑