arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2306.17201cs.CV
收藏

MPM:一种基于掩码姿态建模的统一2D-3D人体姿态表示方法

MPM: A Unified 2D-3D Human Pose Representation via Masked Pose Modeling

Zhejiang University(浙江大学);Key Laboratory of Visual Perception (Zhejiang University), Ministry of Education and Microsoft(视觉感知重点实验室(浙江大学)、教育部与微软);Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security(杭州高新区(滨江)区块链与数据安全研究院);University of Washington(华盛顿大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));ZJU-UIUC Institute, Zhejiang University(浙江大学ZJU-UIUC学院)

Zhenyu Zhang, Wenhao Chai, Zhongyu Jiang, Tian Ye, Mingli Song, Jenq-Neng Hwang, Gaoang Wang

AI总结:

针对2D与3D人体姿态表示未统一的问题,提出基于掩码姿态建模的MPM框架,用单流Transformer和双掩码预训练任务实现多姿态任务统一处理,在MPI-INF-3DHP上达SOTA。

摘要:

近年来,仅从2D人体姿态序列估计3D人体姿态的研究已得到充分探索,但在此之前,尚无研究尝试在共享特征空间中统一2D与3D姿态表示。本文提出MPM(掩码姿态建模)——一种基于掩码姿态建模的统一2D-3D人体姿态表示框架。我们将2D和3D姿态视为类似视觉与语言的两种不同模态,构建了基于Transformer的单流架构。我们采用两个前置任务:掩码2D姿态建模与掩码3D姿态建模对网络进行预训练,再通过全监督方式进一步微调。总掩码率高达71.8%,配合时空掩码采样策略,可提升空间与时间域的关系建模效果。MPM能在单个框架内处理多项任务,包括3D人体姿态估计、遮挡2D姿态下的3D姿态估计以及3D姿态补全。我们在多个广泛使用的人体姿态数据集上开展了大量实验与消融研究,在MPI-INF-3DHP数据集上取得了当前最优性能。

英文摘要:

Estimating 3D human poses only from a 2D human pose sequence is thoroughly explored in recent years. Yet, prior to this, no such work has attempted to unify 2D and 3D pose representations in the shared feature space. In this paper, we propose \mpm, a unified 2D-3D human pose representation framework via masked pose modeling. We treat 2D and 3D poses as two different modalities like vision and language and build a single-stream transformer-based architecture. We apply two pretext tasks, which are masked 2D pose modeling, and masked 3D pose modeling to pre-train our network and use full-supervision to perform further fine-tuning. A high masking ratio of $71.8~\%$ in total with a spatio-temporal mask sampling strategy leads to better relation modeling both in spatial and temporal domains. \mpm~can handle multiple tasks including 3D human pose estimation, 3D pose estimation from occluded 2D pose, and 3D pose completion in a \textbf{single} framework. We conduct extensive experiments and ablation studies on several widely used human pose datasets and achieve state-of-the-art performance on MPI-INF-3DHP.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑