Neural Marionette:体视频运动骨架和潜在动态的无监督学习
Neural Marionette: Unsupervised Learning of Motion Skeleton and Latent Dynamics from Volumetric Video
摘要:
我们提出了神经木偶,一种无监督的方法,从动态序列中发现骨骼结构,并学习生成与观察到的运动动态一致的各种运动。给定任意运动下关节体的点云观察的视频流,我们的方法发现未知的低维骨架关系,可以有效地表示运动。然后利用所发现的结构将动态序列的运动先验编码到潜在结构中,该潜在结构可以被解码为相对关节旋转以表示完整的骨架运动。我们的方法在没有任何关于潜在运动或骨架结构的先验知识的情况下工作,并且我们证明了所发现的结构在表示4D运动序列时甚至可以与手工标记的地面真实骨架相媲美。骨架结构嵌入了可能的运动空间的一般语义,可以为不同的场景生成运动。我们验证了所学习的运动先验可推广到多模态序列生成、两个姿势的插值和运动重定向到不同的骨架结构。
英文摘要:
We present Neural Marionette, an unsupervised approach that discovers the skeletal structure from a dynamic sequence and learns to generate diverse motions that are consistent with the observed motion dynamics. Given a video stream of point cloud observation of an articulated body under arbitrary motion, our approach discovers the unknown low-dimensional skeletal relationship that can effectively represent the movement. Then the discovered structure is utilized to encode the motion priors of dynamic sequences in a latent structure, which can be decoded to the relative joint rotations to represent the full skeletal motion. Our approach works without any prior knowledge of the underlying motion or skeletal structure, and we demonstrate that the discovered structure is even comparable to the hand-labeled ground truth skeleton in representing a 4D sequence of motion. The skeletal structure embeds the general semantics of possible motion space that can generate motions for diverse scenarios. We verify that the learned motion prior is generalizable to the multi-modal sequence generation, interpolation of two poses, and motion retargeting to a different skeletal structure.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。