arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1812.01037cs.CV
收藏

TwoStreamVAN:改进视频生成中的运动建模

TwoStreamVAN: Improving Motion Modeling in Video Generation

Boston University(波士顿大学);University of California Berkeley(加州大学伯克利分校)

Ximeng Sun, Huijuan Xu, Kate Saenko

摘要:

视频生成是一个固有的挑战性任务,因为它需要建模逼真的时间动态以及空间内容。现有方法在单个生成器网络中纠缠运动和内容创建这两个本质上不同的任务,但这种方法难以同时生成合理的运动和内容。为了改进视频生成任务中的运动建模,我们提出了一种双流模型,将运动生成与内容生成分开,称为双流变分对抗网络(TwoStreamVAN)。给定动作标签和噪声向量,我们的模型能够创建清晰和一致的运动,从而产生逼真的视频。其核心思想是逐步生成和融合多尺度运动与其相应的空间内容。我们的模型在标准的Weizmann人类动作,MUG面部表情和VoxCeleb数据集以及我们具有挑战性和复杂运动的各种人类动作的新数据集上显着优于现有方法。我们的代码可在https://github.com/sunxm2357/TwoStreamVAN/上获得。

英文摘要:

Video generation is an inherently challenging task, as it requires modeling realistic temporal dynamics as well as spatial content. Existing methods entangle the two intrinsically different tasks of motion and content creation in a single generator network, but this approach struggles to simultaneously generate plausible motion and content. To im-prove motion modeling in video generation tasks, we propose a two-stream model that disentangles motion generation from content generation, called a Two-Stream Variational Adversarial Network (TwoStreamVAN). Given an action label and a noise vector, our model is able to create clear and consistent motion, and thus yields photorealistic videos. The key idea is to progressively generate and fuse multi-scale motion with its corresponding spatial content. Our model significantly outperforms existing methods on the standard Weizmann Human Action, MUG Facial Expression, and VoxCeleb datasets, as well as our new dataset of diverse human actions with challenging and complex motion. Our code is available at https://github.com/sunxm2357/TwoStreamVAN/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑