arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2209.08790cs.CVcs.LG
收藏

D&D:从动态相机中学习人体动力学

D&D: Learning Human Dynamics from Dynamic Camera

Shanghai Jiao Tong University(上海交通大学);Tencent(腾讯);Qing Yuan Research Institute(清源研究院);AI Institute, Shanghai Jiao Tong University(上海交通大学人工智能研究院);Shanghai Qi Zhi Institute(上海期智研究院)

Jiefeng Li, Siyuan Bian, Chao Xu, Gang Liu, Gang Yu, Cewu Lu

摘要:

从单目视频的3D人体姿态估计最近已经看到了显著的改进。然而,大多数最先进的方法都是基于运动学的,这容易产生具有明显伪影的物理上难以置信的运动。目前基于动力学的方法可以预测物理上合理的运动,但仅限于简单的场景与静态相机视图。在这项工作中,我们提出了D&D(从动态摄像机学习人体动态),它利用物理定律从移动摄像机的野外视频中重建3D人体运动。D&D引入了惯性力控制(IFC),通过考虑动态相机的惯性力来解释非惯性局部框架中的3D人体运动。为了学习有限注释的地面接触,我们开发了概率接触扭矩(PCT),它是通过接触概率的可微分采样计算的,并用于生成运动。接触状态可以通过鼓励模型生成正确的运动来进行弱监督。此外,我们提出了一个细心的PD控制器,调整目标的姿态状态,使用时间信息,以获得平滑和准确的姿态控制。我们的方法是完全基于神经的,无需离线优化或物理引擎模拟即可运行。大规模的三维人体运动基准的实验证明了D&D的有效性,在那里我们表现出优越的性能对最先进的运动学为基础的和基于动力学的方法。代码可在https://github.com/Jeffsjtu/DnD上获得

英文摘要:

3D human pose estimation from a monocular video has recently seen significant improvements. However, most state-of-the-art methods are kinematics-based, which are prone to physically implausible motions with pronounced artifacts. Current dynamics-based methods can predict physically plausible motion but are restricted to simple scenarios with static camera view. In this work, we present D&D (Learning Human Dynamics from Dynamic Camera), which leverages the laws of physics to reconstruct 3D human motion from the in-the-wild videos with a moving camera. D&D introduces inertial force control (IFC) to explain the 3D human motion in the non-inertial local frame by considering the inertial forces of the dynamic camera. To learn the ground contact with limited annotations, we develop probabilistic contact torque (PCT), which is computed by differentiable sampling from contact probabilities and used to generate motions. The contact state can be weakly supervised by encouraging the model to generate correct motions. Furthermore, we propose an attentive PD controller that adjusts target pose states using temporal information to obtain smooth and accurate pose control. Our approach is entirely neural-based and runs without offline optimization or simulation in physics engines. Experiments on large-scale 3D human motion benchmarks demonstrate the effectiveness of D&D, where we exhibit superior performance against both state-of-the-art kinematics-based and dynamics-based methods. Code is available at https://github.com/Jeffsjtu/DnD

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑