arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2102.09737cs.CVcs.SDeess.ASeess.IV
收藏

单镜头音频到动画视频生成

One Shot Audio to Animated Video Generation

Hike Private Limited(Hike私人有限公司);IIT Delhi(印度理工学院德里分校)

Neeraj Kumar, Srishti Goel, Ankur Narang, Brejesh Lall, Mujtaba Hasan, Pranshu Agarwal, Dipankar Sarkar

摘要:

我们认为具有挑战性的问题,音频动画视频生成。我们提出了一种新的方法OneShotAu 2AV生成一个动画视频的任意长度使用一个音频剪辑和一个人的一个单一的看不见的图像作为输入。所提出的方法包括两个阶段。在第一阶段中,OneShotAu 2AV在给定音频和人物图像的情况下生成人类领域中的说话头部视频。在第二阶段中,将来自人类领域的讲话头部视频转换为动画领域。第一阶段的模型架构包括基于空间自适应归一化的多级生成器和多个多级鉴别器以及多个对抗性和非对抗性损失。第二阶段利用基于注意力的标准化驱动的GAN架构以及基于时间预测器的再循环损失和与唇同步损失耦合的闪烁损失,用于动画视频的无监督生成。在我们的方法中,输入的音频片段不限于任何特定的语言,这使得该方法多语言的适用性。OneShotAu 2AV可以生成动画视频,这些视频具有:(a)与音频同步的嘴唇运动,(b)自然的面部表情,如眨眼和眉毛运动,(c)头部运动。实验评估表明,OneShotAu 2AV在多个定量指标上的性能优于U-GAT-IT和MoncleGan,这些指标包括KID(内核初始距离)、字出错率、闪烁/秒

英文摘要:

We consider the challenging problem of audio to animated video generation. We propose a novel method OneShotAu2AV to generate an animated video of arbitrary length using an audio clip and a single unseen image of a person as an input. The proposed method consists of two stages. In the first stage, OneShotAu2AV generates the talking-head video in the human domain given an audio and a person's image. In the second stage, the talking-head video from the human domain is converted to the animated domain. The model architecture of the first stage consists of spatially adaptive normalization based multi-level generator and multiple multilevel discriminators along with multiple adversarial and non-adversarial losses. The second stage leverages attention based normalization driven GAN architecture along with temporal predictor based recycle loss and blink loss coupled with lipsync loss, for unsupervised generation of animated video. In our approach, the input audio clip is not restricted to any specific language, which gives the method multilingual applicability. OneShotAu2AV can generate animated videos that have: (a) lip movements that are in sync with the audio, (b) natural facial expressions such as blinks and eyebrow movements, (c) head movements. Experimental evaluation demonstrates superior performance of OneShotAu2AV as compared to U-GAT-IT and RecycleGan on multiple quantitative metrics including KID(Kernel Inception Distance), Word error rate, blinks/sec

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑