arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2104.11216cs.CVcs.AIcs.LGstat.ML
收藏

基于运动程序的分层运动理解

Hierarchical Motion Understanding via Motion Programs

Stanford University(斯坦福大学);MIT(麻省理工学院)

Sumith Kulal, Jiayuan Mao, Alex Aiken, Jiajun Wu

摘要:

目前的人体运动视频分析方法主要集中在原始像素或关键点作为推理的基本单位。我们认为,添加更高级别的运动原语,可以捕捉自然的粗糙运动单元,如后摆或后续通过,可以用来改善下游的分析任务。这种更高级别的抽象还可以捕获关键特征,例如重复原语的循环,这些特征目前在较低级别的表示中无法访问。因此,我们引入运动程序,神经符号,程序一样的表示,表示运动作为一个组成的高级原语。我们还提出了一个系统,用于自动诱导人体运动的视频和利用运动程序在视频合成的运动程序。实验表明,运动程序可以准确地描述一组不同的人类运动和推断程序包含语义有意义的运动原语,如手臂摆动和跳跃千斤顶。我们的表示也有利于下游任务,如视频插值和视频预测,并优于现成的模型。我们进一步展示了这些程序如何检测各种重复运动,并促进交互式视频编辑。

英文摘要:

Current approaches to video analysis of human motion focus on raw pixels or keypoints as the basic units of reasoning. We posit that adding higher-level motion primitives, which can capture natural coarser units of motion such as backswing or follow-through, can be used to improve downstream analysis tasks. This higher level of abstraction can also capture key features, such as loops of repeated primitives, that are currently inaccessible at lower levels of representation. We therefore introduce Motion Programs, a neuro-symbolic, program-like representation that expresses motions as a composition of high-level primitives. We also present a system for automatically inducing motion programs from videos of human motion and for leveraging motion programs in video synthesis. Experiments show that motion programs can accurately describe a diverse set of human motions and the inferred programs contain semantically meaningful motion primitives, such as arm swings and jumping jacks. Our representation also benefits downstream tasks such as video interpolation and video prediction and outperforms off-the-shelf models. We further demonstrate how these programs can detect diverse kinds of repetitive motion and facilitate interactive video editing.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑