iPOKE:用于受控随机视频合成的戳静止图像
iPOKE: Poking a Still Image for Controlled Stochastic Video Synthesis
摘要:
一个静态的场景会对一个局部的戳有什么反应?如果你能局部地推动一个物体,那么它的其他部分会受到什么影响呢?尽管我们世界的随机性造成了明显的变化,但仍会有独特的运动。这些结果是由物体的特征运动学决定的,这些运动学决定了由局部相互作用引起的物体的整体运动。相反,物体的运动提供了关于其基本的独特运动学及其部件之间的相互依赖性的关键信息。这种双向关系促使学习对象运动学和合理的未来图像序列之间的双射映射。因此,我们提出了iPOKE -可逆预测的对象Kinetics-,条件的初始帧和本地戳,允许采样对象的运动学,并建立一个一对一的对应关系,相应的似是而非的视频,从而提供了一个受控的随机视频合成。与以前的作品相比,我们不会生成任意逼真的视频,而是提供有效的运动控制,同时仍然捕捉我们环境的随机性和它所带来的合理结果的多样性。此外,我们的方法可以转移到新的对象实例的运动学,并不局限于特定的对象类。我们的项目页面可在https://bit.ly/3dJN4Lf上找到。
英文摘要:
How would a static scene react to a local poke? What are the effects on other parts of an object if you could locally push it? There will be distinctive movement, despite evident variations caused by the stochastic nature of our world. These outcomes are governed by the characteristic kinematics of objects that dictate their overall motion caused by a local interaction. Conversely, the movement of an object provides crucial information about its underlying distinctive kinematics and the interdependencies between its parts. This two-way relation motivates learning a bijective mapping between object kinematics and plausible future image sequences. Therefore, we propose iPOKE -- invertible Prediction of Object Kinematics -- that, conditioned on an initial frame and a local poke, allows to sample object kinematics and establishes a one-to-one correspondence to the corresponding plausible videos, thereby providing a controlled stochastic video synthesis. In contrast to previous works, we do not generate arbitrary realistic videos, but provide efficient control of movements, while still capturing the stochastic nature of our environment and the diversity of plausible outcomes it entails. Moreover, our approach can transfer kinematics onto novel object instances and is not confined to particular object classes. Our project page is available at https://bit.ly/3dJN4Lf.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。