Modeling humans in physical scenes is vital for understanding human-environment interactions for applications involving augmented reality or assessment of human actions from video (e.g. sports or physical rehabilitation). State-of-the-art literature begins with a 3D human pose, from monocular or multiple views, and uses this representation to ground the person within a 3D world space. While standard metrics for accuracy capture joint position errors, they do not consider physical plausibility of the 3D pose. This limitation has motivated researchers to propose other metrics evaluating jitter, floor penetration, and unbalanced postures. Yet, these approaches measure independent instances of errors and are not representative of balance or stability during motion. In this work, we propose measuring physical plausibility from within physics simulation. We introduce two metrics to capture the physical plausibility and stability of predicted 3D poses from any 3D Human Pose Estimation model. Using physics simulation, we discover correlations with existing plausibility metrics and measuring stability during motion. We evaluate and compare the performances of two state-of-the-art methods, a multi-view triangulated baseline, and ground truth 3D markers from the Human3.6m dataset.
The rapid development of multi-view 3D human pose estimation (HPE) is attributed to the maturation of monocular 2D HPE and the geometry of 3D reconstruction. However, 2D detection outliers in occluded views due to neglect of view consistency, and 3D implausible poses due to lack of pose coherence, remain challenges. To solve this, we introduce a Multi-View Fusion module to refine 2D results by establishing view correlations. Then, Holistic Triangulation is proposed to infer the whole pose as an entirety, and anatomy prior is injected to maintain the pose coherence and improve the plausibility.…
National University of Singapore（新加坡国立大学）；Huawei International Pte Ltd（华为国际私人有限公司）
Existing 3D human pose estimators suffer poor generalization performance to new datasets, largely due to the limited diversity of 2D-3D pose pairs in the training data. To address this problem, we present PoseAug, a new auto-augmentation framework that learns to augment the available training poses towards a greater diversity and thus improve generalization of the trained 2D-to-3D pose estimator. Specifically, PoseAug introduces a novel pose augmentor that learns to adjust various geometry factors (e.g., posture, body size, view point and position) of a pose through differentiable operations. …
Beihang University（北京航空航天大学）；SenseTime Research（商汤科技研究院）；The University of Hong Kong（香港大学）；Shanghai AI Laboratory（上海人工智能实验室）
Human pose estimation aims to accurately estimate a wide variety of human poses. However, existing datasets often follow a long-tailed distribution that unusual poses only occupy a small portion, which further leads to the lack of diversity of rare poses. These issues result in the inferior generalization ability of current pose estimators. In this paper, we present a simple yet effective data augmentation method, termed Pose Transformation (PoseTrans), to alleviate the aforementioned problems. Specifically, we propose Pose Transformation Module (PTM) to create new training samples that have d…
Improving 2D Human Pose Estimation in Rare Camera Views with Synthetic Data
Czech Technical University in Prague（布拉格捷克技术大学）；Faculty of Electrical Engineering（电气工程学院）；Department of Cybernetics（控制论系）；Visual Recognition Group（视觉识别组）
TU Wien（维也纳工业大学）；University of Technology Sydney（悉尼科技大学）
Observational noise, inaccurate segmentation and ambiguity due to symmetry and occlusion lead to inaccurate object pose estimates. While depth- and RGB-based pose refinement approaches increase the accuracy of the resulting pose estimates, they are susceptible to ambiguity in the observation as they consider visual alignment. We propose to leverage the fact that we often observe static, rigid scenes. Thus, the objects therein need to be under physically plausible poses. We show that considering plausibility reduces ambiguity and, in consequence, allows poses to be more accurately predicted in …
Training accurate 3D human pose estimators requires large amount of 3D ground-truth data which is costly to collect. Various weakly or self supervised pose estimation methods have been proposed due to lack of 3D data. Nevertheless, these methods, in addition to 2D ground-truth poses, require either additional supervision in various forms (e.g. unpaired 3D ground truth data, a small subset of labels) or the camera parameters in multiview settings. To address these problems, we present EpipolarPose, a self-supervised learning method for 3D human pose estimation, which does not need any 3D ground…
University of California, Los Angeles (UCLA)（加州大学洛杉矶分校）；International Center for AI and Robot Autonomy (CARA)（国际人工智能与机器人自主中心）
We propose a new 3D holistic++ scene understanding problem, which jointly tackles two tasks from a single-view image: (i) holistic scene parsing and reconstruction---3D estimations of object bounding boxes, camera pose, and room layout, and (ii) 3D human pose estimation. The intuition behind is to leverage the coupled nature of these two tasks to improve the granularity and performance of scene understanding. We propose to exploit two critical and essential connections between these two tasks: (i) human-object interaction (HOI) to model the fine-grained relations between agents and objects in …
StarPose: 3D Human Pose Estimation via Spatial-Temporal Autoregressive Diffusion
School of Computer Science and Engineering, South China University of Technology（华南理工大学计算机科学与工程学院）；Centre for Smart Health, The Hong Kong Polytechnic University（香港理工大学智能健康中心）；Cloud Computing Center, Chinese Academy of Sciences（中国科学院云计算中心）；Guangzhou Yichuang Information Technology Co., Ltd.（广州亿创信息技术有限公司）；School of Computing and Information Systems, Singapore Management University（新加坡管理大学计算机与信息系）
Accurate estimation of 3D human motion from monocular video requires modeling both kinematics (body motion without physical forces) and dynamics (motion with physical forces). To demonstrate this, we present SimPoE, a Simulation-based approach for 3D human Pose Estimation, which integrates image-based kinematic inference and physics-based dynamics modeling. SimPoE learns a policy that takes as input the current-frame pose estimate and the next image frame to control a physically-simulated character to output the next-frame pose estimate. The policy contains a learnable kinematic pose refinemen…
Understanding the 3D world from 2D images involves more than detection and segmentation of the objects within the scene. It also includes the interpretation of the structure and arrangement of the scene elements. Such understanding is often rooted in recognizing the physical world and its limitations, and in prior knowledge as to how similar typical scenes are arranged. In this research we pose a new challenge for neural network (or other) scene understanding algorithms - can they distinguish between plausible and implausible scenes? Plausibility can be defined both in terms of physical proper…
PAMD: Plausibility-Aware Motion Diffusion Model for Long Dance Generation
School of Computer Science and Engineering, Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Ministry of Education, Southeast University（计算机科学与工程学院，新一代人工智能技术及交叉应用重点实验室，教育部，东南大学）；State Key Laboratory of Media Convergence and Communication, Communication University of China（媒体融合与传播国家重点实验室，中国传媒大学）；School of Computer Science and Software Engineering, National Engineering Laboratory for Big Data System Computing Technology, Guangdong Key Laboratory of Intelligent Information Processing, Shenzhen University（计算机科学与软件工程学院，大数据系统计算技术国家工程实验室，广东智能信息处理重点实验室，深圳大学）；School of Computer Science and Engineering, Nanjing University of Science and Technology（计算机科学与工程学院，南京理工大学）
Izmir Institute of Technology（伊兹密尔理工学院）；Instituto de Investigación en Ingeniería de Aragón (I3A)（阿拉贡工程研究院（I3A））；Universidad de Zaragoza（萨拉戈萨大学）
Any city-scale visual localization system has to overcome long-term appearance changes, such as varying illumination conditions or seasonal changes between query and database images. Since semantic content is more robust to such changes, we exploit semantic information to improve visual localization. In our scenario, the database consists of gnomonic views generated from panoramic images (e.g. Google Street View) and query images are collected with a standard field-of-view camera at a different time. To improve localization, we check the semantic similarity between query and database images, w…
This paper is about extremely robust and lightweight localisation using LiDAR point clouds based on instance segmentation and graph matching. We model 3D point clouds as fully-connected graphs of semantically identified components where each vertex corresponds to an object instance and encodes its shape. Optimal vertex association across graphs allows for full 6-Degree-of-Freedom (DoF) pose estimation and place recognition by measuring similarity. This representation is very concise, condensing the size of maps by a factor of 25 against the state-of-the-art, requiring only 3kB to represent a 1…
Existing deep models predict 2D and 3D kinematic poses from video that are approximately accurate, but contain visible errors that violate physical constraints, such as feet penetrating the ground and bodies leaning at extreme angles. In this paper, we present a physics-based method for inferring 3D human motion from video sequences that takes initial 2D and 3D pose estimates as input. We first estimate ground contact timings with a novel prediction network which is trained without hand-labeled data. A physics-based trajectory optimization then solves for a physically-plausible motion, based o…
TU Wien（维也纳工业大学）；Automation and Control Institute（自动化与控制研究所）
Accurate and robust object pose estimation for robotics applications requires verification and refinement steps. In this work, we propose to integrate hypotheses verification with object pose refinement guided by physics simulation. This allows the physical plausibility of individual object pose estimates and the stability of the estimated scene to be considered in a unified optimization. The proposed method is able to adapt to scenes of multiple objects and efficiently focuses on refining the most promising object poses in multi-hypotheses scenarios. We call this integrated approach VeREFINE …
Imperial College London（帝国理工学院）；Niantic, Inc.（Niantic公司）；Graz Uni. of Technology（格拉茨工业大学）；Rokid Corp. Ltd.（Rokid公司）；HUST（华中科技大学）；Megvii Research Nanjing（旷视南京研究院）；SUNY Buffalo（纽约州立大学布法罗分校）；BUPT（北京邮电大学）；Uni. of West Bohemia（西波希米亚大学）；Fudan Uni.（复旦大学）；TUM（慕尼黑工业大学）；Uni. of Bonn（波恩大学）；NUS（新加坡国立大学）；Harbin Inst. of Technology（哈尔滨工业大学）；CUHK（香港中文大学）；ETH Zurich（苏黎世联邦理工学院）；NVIDIA Research（英伟达研究院）；NAVER LABS Europe（NAVER欧洲实验室）；ENPC ParisTech（巴黎高科路桥学校）；UNIST（蔚山科学技术院）；KAIST（韩国科学技术院）
We study how well different types of approaches generalise in the task of 3D hand pose estimation under single hand scenarios and hand-object interaction. We show that the accuracy of state-of-the-art methods can drop, and that they fail mostly on poses absent from the training set. Unfortunately, since the space of hand poses is highly dimensional, it is inherently not feasible to cover the whole space densely, despite recent efforts in collecting large-scale training datasets. This sampling problem is even more severe when hands are interacting with objects and/or inputs are RGB rather than …
Google LLC（谷歌有限责任公司）；Calico Life Sciences LLC（Calico生命科学有限责任公司）
We present a method to infer the 3D pose of mice, including the limbs and feet, from monocular videos. Many human clinical conditions and their corresponding animal models result in abnormal motion, and accurately measuring 3D motion at scale offers insights into health. The 3D poses improve classification of health-related attributes over 2D representations. The inferred poses are accurate enough to estimate stride length even when the feet are mostly occluded. This method could be applied as part of a continuous monitoring system to non-invasively measure animal health.
In this paper, we deal with two challenges for measuring the similarity of the subject identities in practical video-based face recognition - the variation of the head pose in uncontrolled environments and the computational expense of processing videos. Since the frame-wise feature mean is unable to characterize the pose diversity among frames, we define and preserve the overall pose diversity and closeness in a video. Then, identity will be the only source of variation across videos since the pose varies even within a single video. Instead of simply using all the frames, we select those faces…
Markerless Stride Length estimation in Athletic using Pose Estimation with monocular vision
Dipartimento di Ingegneria dell’Innovazione Università del Salento（创新工程学院，萨勒诺大学）；Istituto di Scienze Applicate e sIstemi Intelligenti - ISASI Consiglio Nazionale delle Ricerche - (CNR)（应用科学与智能系统研究所 - 国家研究理事会 - (CNR)）；Consiglio Nazionale delle Ricerche - (CNR)（国家研究理事会 - (CNR)）
University of Toronto（多伦多大学）；Vector Institute（向量研究所）；Nvidia（英伟达）
Human motion synthesis is an important problem with applications in graphics, gaming and simulation environments for robotics. Existing methods require accurate motion capture data for training, which is costly to obtain. Instead, we propose a framework for training generative models of physically plausible human motion directly from monocular RGB videos, which are much more widely available. At the core of our method is a novel optimization formulation that corrects imperfect image-based pose estimations by enforcing physics constraints and reasons about contacts in a differentiable way. This…
PhysPose: Refining 6D Object Poses with Physical Constraints
Czech Institute of Informatics, Robotics and Cybernetics, Czech Technical University in Prague（布拉格捷克理工大学捷克信息学、机器人学与控制论学院）；Faculty of Electrical Engineering, Czech Technical University in Prague（布拉格捷克理工大学电气工程学院）；Inria - Département d’Informatique de l’École normale supérieure, PSL Research University（法国国家信息与自动化研究所 - 巴黎高等师范学院信息学系，PSL大学）
Physics-Informed Diffusion for Biomechanically Plausible 3D Sign Language Generation
Department of Computer Science, University of Bari Aldo Moro（巴里阿尔多·莫罗大学计算机科学系）；Institute for Technological Development and Innovation in Communications, University of Las Palmas de Gran Canaria（拉斯帕尔马斯大加那利大学通信技术发展与创新研究所）
Physically Plausible Human-Object Rendering from Sparse Views via 3D Gaussian Splatting
College of Computer Science, Zhejiang University（浙江大学计算机科学与技术学院）；Department of Computer Science and Engineering, The Hong Kong University of Science and Technology（香港科技大学计算机科学与工程系）
Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Facebook Reality Labs（Facebook现实实验室）
Marker-less 3D human motion capture from a single colour camera has seen significant progress. However, it is a very challenging and severely ill-posed problem. In consequence, even the most accurate state-of-the-art approaches have significant limitations. Purely kinematic formulations on the basis of individual joints or skeletons, and the frequent frame-wise reconstruction in state-of-the-art methods greatly limit 3D accuracy and temporal stability compared to multi-view or marker-based motion capture. Further, captured 3D poses are often physically incorrect and biomechanically implausible…
Measuring proximity to standard planes during fetal brain ultrasound scanning
Wellcome/EPSRC center for Interventional and Surgical Sciences (WEISS)（Wellcome/EPSRC介入与外科科学中心（WEISS））；University College London (UCL)（伦敦大学学院（UCL））；Dept of Computer Science, UCL（伦敦大学学院计算机科学系）；Dept of Obstetrics and Gynecology, University Hospital of Rennes（雷恩大学医院妇产科）；Univ Rennes, INSERM, LTSI - UMR 1099（雷恩大学、法国国家健康与医学研究院、图像与信号处理实验室 - UMR 1099）；Fetal Medicine Unit, Elizabeth Garrett Anderson Wing, UCLH NHS Foundation Trust（伦敦大学学院医院NHS基金会信托伊丽莎白·加勒特·安德森翼胎儿医学科）；Elizabeth Garrett Anderson Institute for Women’s Health, UCL（伦敦大学学院伊丽莎白·加勒特·安德森妇女健康研究所）；Dept of Obstetrics and Gynecology, UCLH（伦敦大学学院医院妇产科）
该研究提出一种半监督分割模型结合6D平面姿态回归的流水线，在NVIDIA Clara AGX边缘设备上实现实时接近度检测，可辅助胎儿脑部超声扫描的导航支持。
