arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2212.07413cs.CV
收藏

走向平滑的视频合成

Towards Smooth Video Composition

The Chinese University of Hong Kong(香港中文大学);Shanghai AI Laboratory(上海人工智能实验室);Ant Group(蚂蚁集团);University of California, Los Angeles(加利福尼亚大学洛杉矶分校)

Qihang Zhang, Ceyuan Yang, Yujun Shen, Yinghao Xu, Bolei Zhou

摘要:

视频生成需要随着时间的推移合成具有动态内容的一致且持久的帧。这项工作研究了使用生成对抗网络(GANs)构建任意长度(从几帧到无限长)视频的时间关系。首先,对组成相邻帧,我们表明,单图像生成的无混叠操作,加上充分的预先学习的知识,带来了平稳的帧过渡,而不影响每帧的质量。其次,通过将时间移位模块(TSM),最初设计用于视频理解,到的同步,我们设法推进生成器在合成更一致的动态。第三,我们开发了一种新的基于B样条的运动表示,以确保时间平滑,实现无限长的视频生成。它可以超出训练中使用的帧数。还提出了一种低秩时间调制,以减轻长视频生成的重复内容。我们在各种数据集上评估了我们的方法,并在视频生成基线上显示了实质性的改进。代码和模型将在https://genforce.github.io/StyleSV上公开。

英文摘要:

Video generation requires synthesizing consistent and persistent frames with dynamic content over time. This work investigates modeling the temporal relations for composing video with arbitrary length, from a few frames to even infinite, using generative adversarial networks (GANs). First, towards composing adjacent frames, we show that the alias-free operation for single image generation, together with adequately pre-learned knowledge, brings a smooth frame transition without compromising the per-frame quality. Second, by incorporating the temporal shift module (TSM), originally designed for video understanding, into the discriminator, we manage to advance the generator in synthesizing more consistent dynamics. Third, we develop a novel B-Spline based motion representation to ensure temporal smoothness to achieve infinite-length video generation. It can go beyond the frame number used in training. A low-rank temporal modulation is also proposed to alleviate repeating contents for long video generation. We evaluate our approach on various datasets and show substantial improvements over video generation baselines. Code and models will be publicly available at https://genforce.github.io/StyleSV.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑