arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.12959cs.CVcs.AIcs.GRcs.LGcs.RO
收藏

WHAC:世界坐标系下的人体与相机

WHAC: World-grounded Humans and Cameras

SenseTime Research(商汤科技研究院);The University of Tokyo(东京大学);Nanyang Technological University(南洋理工大学);Shanghai AI Laboratory(上海人工智能实验室)

Wanqi Yin, Zhongang Cai, Ruisi Wang, Fanzhou Wang, Chen Wei, Haiyi Mei, Weiye Xiao, Zhitao Yang, Qingping Sun, Atsushi Yamashita, Ziwei Liu, Lei Yang

AI总结:

提出WHAC框架,利用人体运动提供的绝对空间线索与相机坐标系下的深度估计,联合恢复世界坐标系下SMPL-X人体模型与相机姿态,并引入合成数据集WHAC-A-Mole,实验验证其优越性。

摘要:

从单目视频中估计具有准确尺度、位于世界坐标系下的人体与相机轨迹,是一个极具吸引力但又充满挑战且不适定(ill-posed)的问题。在本研究中,我们旨在联合恢复具有表现力的参数化人体模型(即SMPL-X)及相应的相机姿态,通过利用三个关键要素之间的协同作用:世界、人体与相机。我们的方法基于两个关键观察。首先,相机坐标系下的SMPL-X估计方法能够直接恢复绝对人体深度。其次,人体运动本身提供了绝对空间线索。通过整合这些见解,我们提出了一种新颖框架,称为WHAC,以促进世界坐标系下具有表现力的人体姿态与形状估计(EHPS)以及相机姿态估计,且无需依赖传统优化技术。此外,我们提出了一个新的合成数据集WHAC-A-Mole,其中包含精确标注的人体与相机,并具有多样化的交互式人体运动以及真实的相机轨迹。在标准基准和新建立的基准上进行的大量实验,凸显了我们框架的优越性和有效性。我们将公开代码和数据集。

英文摘要:

Estimating human and camera trajectories with accurate scale in the world coordinate system from a monocular video is a highly desirable yet challenging and ill-posed problem. In this study, we aim to recover expressive parametric human models (i.e., SMPL-X) and corresponding camera poses jointly, by leveraging the synergy between three critical players: the world, the human, and the camera. Our approach is founded on two key observations. Firstly, camera-frame SMPL-X estimation methods readily recover absolute human depth. Secondly, human motions inherently provide absolute spatial cues. By integrating these insights, we introduce a novel framework, referred to as WHAC, to facilitate world-grounded expressive human pose and shape estimation (EHPS) alongside camera pose estimation, without relying on traditional optimization techniques. Additionally, we present a new synthetic dataset, WHAC-A-Mole, which includes accurately annotated humans and cameras, and features diverse interactive human motions as well as realistic camera trajectories. Extensive experiments on both standard and newly established benchmarks highlight the superiority and efficacy of our framework. We will make the code and dataset publicly available.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑