点击移动:用稀疏运动控制视频生成
Click to Move: Controlling Video Generation with Sparse Motion
摘要:
本文介绍了点击移动(C2M),一种新的视频生成框架,用户可以通过鼠标点击指定场景中的关键对象的简单对象轨迹来控制合成视频的运动。我们的模型接收作为输入的初始帧,其相应的分割图和稀疏的运动矢量编码的输入由用户提供。它从给定帧开始输出一个看似合理的视频序列,并具有与用户输入一致的运动。值得注意的是,我们提出的深度架构采用了一个图形卷积网络(GCN),以整体的方式对场景中所有对象的运动进行建模,并有效地结合了稀疏的用户运动信息和图像特征。实验结果表明,C2M优于现有的方法在两个公开的数据集,从而证明了我们的GCN框架在建模对象的相互作用的有效性。源代码可在https://github.com/PierfrancescoArdino/C2M上公开获得。
英文摘要:
This paper introduces Click to Move (C2M), a novel framework for video generation where the user can control the motion of the synthesized video through mouse clicks specifying simple object trajectories of the key objects in the scene. Our model receives as input an initial frame, its corresponding segmentation map and the sparse motion vectors encoding the input provided by the user. It outputs a plausible video sequence starting from the given frame and with a motion that is consistent with user input. Notably, our proposed deep architecture incorporates a Graph Convolution Network (GCN) modelling the movements of all the objects in the scene in a holistic manner and effectively combining the sparse user motion information and image features. Experimental results show that C2M outperforms existing methods on two publicly available datasets, thus demonstrating the effectiveness of our GCN framework at modelling object interactions. The source code is publicly available at https://github.com/PierfrancescoArdino/C2M.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。