Markerless pose estimation allows reconstructing human movement from multiple synchronized and calibrated views, and has the potential to make movement analysis easy and quick, including gait analysis. This could enable much more frequent and quantitative characterization of gait impairments, allowing better monitoring of outcomes and responses to interventions. However, the impact of different keypoint detectors and reconstruction algorithms on markerless pose estimation accuracy has not been thoroughly evaluated. We tested these algorithmic choices on data acquired from a multicamera system from a heterogeneous sample of 25 individuals seen in a rehabilitation hospital. We found that using a top-down keypoint detector and reconstructing trajectories with an implicit function enabled accurate, smooth and anatomically plausible trajectories, with a noise in the step width estimates compared to a GaitRite walkway of only 8mm.
We present a method to combine markerless motion capture and dense pose feature estimation into a single framework. We demonstrate that dense pose information can help for multiview/single-view motion capture, and multiview motion capture can help the collection of a high-quality dataset for training the dense pose detector. Specifically, we first introduce a novel markerless motion capture method that can take advantage of dense parsing capability provided by the dense pose detector. Thanks to the introduced dense human parsing ability, our method is demonstrated much more efficient, and accu…
Human pose estimation is a very active research field, stimulated by its important applications in robotics, entertainment or health and sports sciences, among others. Advances in convolutional networks triggered noticeable improvements in 2D pose estimation, leading modern 3D markerless motion capture techniques to an average error per joint of 20 mm. However, with the proliferation of methods, it is becoming increasingly difficult to make an informed choice. Here, we review the leading human pose estimation methods of the past five years, focusing on metrics, benchmarks and method structures…
We propose a CNN-based approach for multi-camera markerless motion capture of the human body. Unlike existing methods that first perform pose estimation on individual cameras and generate 3D models as post-processing, our approach makes use of 3D reasoning throughout a multi-stage approach. This novelty allows us to use provisional 3D models of human pose to rethink where the joints should be located in the image and to recover from past mistakes. Our principled refinement of 3D human poses lets us make use of image cues, even from images where we previously misdetected joints, to refine our e…
Solving the camera-to-robot pose is a fundamental requirement for vision-based robot control, and is a process that takes considerable effort and cares to make accurate. Traditional approaches require modification of the robot via markers, and subsequent deep learning approaches enabled markerless feature extraction. Mainstream deep learning methods only use synthetic data and rely on Domain Randomization to fill the sim-to-real gap, because acquiring the 3D annotation is labor-intensive. In this work, we go beyond the limitation of 3D annotations for real-world data. We propose an end-to-end …
Traditional approaches to extrinsic calibration use fiducial markers and learning-based approaches rely heavily on simulation data. In this work, we present a learning-based markerless extrinsic calibration system that uses a depth camera and does not rely on simulation data. We learn models for end-effector (EE) segmentation, single-frame rotation prediction and keypoint detection, from automatically generated real-world data. We use a transformation trick to get EE pose estimates from rotation predictions and a matching algorithm to get EE pose estimates from keypoint predictions. We further…
MLRecon: Robust Markerless Freehand 3D Ultrasound Reconstruction via Coarse-to-Fine Pose Estimation
Institute of Biomedical Manufacturing and Life Quality Engineering, School of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai, China（生物医学制造与生命质量工程学院，机械工程学院，上海交通大学，上海，中国）；Department of Ultrasound in Medicine, Shanghai Sixth People's Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai, China（医学超声科，上海第六人民医院（隶属于上海交通大学医学院））；Institute of Medical Robotics, Shanghai Jiao Tong University, Shanghai, China（医学机器人研究院，上海交通大学，上海，中国）
Suture needle localization is necessary for autonomous suturing. Previous approaches in autonomous suturing often relied on fiducial markers rather than markerless detection schemes for localizing a suture needle due to the inconsistency of markerless detections. However, fiducial markers are not practical for real-world applications and can often be occluded from environmental factors in surgery (e.g., blood). Therefore in this work, we present a robust tracking approach for estimating the 6D pose of a suture needle when using inconsistent detections. We define observation models based on sut…
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Microsoft（微软）；Body Labs Inc.（Body Labs公司）；UC Berkeley（加州大学伯克利分校）；University of Würzburg（维尔茨堡大学）；Australian National University（澳大利亚国立大学）
Existing marker-less motion capture methods often assume known backgrounds, static cameras, and sequence specific motion priors, which narrows its application scenarios. Here we propose a fully automatic method that given multi-view video, estimates 3D human motion and body shape. We take recent SMPLify \cite{bogo2016keep} as the base method, and extend it in several ways. First we fit the body to 2D features detected in multi-view images. Second, we use a CNN method to segment the person in each image and fit the 3D body model to the contours to further improves accuracy. Third we utilize a g…
In this paper we present an approach for learning to imitate human behavior on a semantic level by markerless visual observation. We analyze a set of spatial constraints on human pose data extracted using convolutional pose machines and object informations extracted from 2D image sequences. A scene analysis, based on an ontology of objects and affordances, is combined with continuous human pose estimation and spatial object relations. Using a set of constraints we associate the observed human actions with a set of executable robot commands. We demonstrate our approach in a kitchen task, where …
Markerless Stride Length estimation in Athletic using Pose Estimation with monocular vision
Dipartimento di Ingegneria dell’Innovazione Università del Salento（创新工程学院，萨勒诺大学）；Istituto di Scienze Applicate e sIstemi Intelligenti - ISASI Consiglio Nazionale delle Ricerche - (CNR)（应用科学与智能系统研究所 - 国家研究理事会 - (CNR)）；Consiglio Nazionale delle Ricerche - (CNR)（国家研究理事会 - (CNR)）
University of Konstanz（康斯坦茨大学）；Max Planck Institute of Animal Behavior（马克斯·普朗克动物行为研究所）；Technische Universität München（慕尼黑工业大学）；Eötvös Loránd University（厄特沃什·罗兰大学）；Hungarian Academy of Sciences（匈牙利科学院）
Recent advances in machine learning and computer vision are revolutionizing the field of animal behavior by enabling researchers to track the poses and locations of freely moving animals without any marker attachment. However, large datasets of annotated images of animals for markerless pose tracking, especially high-resolution images taken from multiple angles with accurate 3D annotations, are still scant. Here, we propose a method that uses a motion capture (mo-cap) system to obtain a large amount of annotated data on animal movement and posture (2D and 3D) in a semi-automatic manner. Our me…
Template-based 3D object tracking still lacks a high-precision benchmark of real scenes due to the difficulty of annotating the accurate 3D poses of real moving video objects without using markers. In this paper, we present a multi-view approach to estimate the accurate 3D poses of real moving objects, and then use binocular data to construct a new benchmark for monocular textureless 3D object tracking. The proposed method requires no markers, and the cameras only need to be synchronous, relatively fixed as cross-view and calibrated. Based on our object-centered model, we jointly optimize the …
VideoRun2D Demo: Markerless Body Tracking for Biomechanical Analysis of Running
BiometricsAI, Universidad Autonoma de Madrid (UAM)（生物识别AI中心，马德里自治大学（UAM））；Biomechanics Lab, Universidad Politecnica de Madrid (UPM)（生物力学实验室，马德里理工大学（UPM））
Markerless Motion Capture for Biomechanical Whole-Body Kinematic Estimation in Infants
Center for Bionic Medicine, Shirley Ryan AbilityLab（生物医学中心，Shirley Ryan AbilityLab）；Dept. of Physical Therapy and Human Movement Science, Northwestern University（物理治疗与人类运动科学系，西北大学）；Dept. of Biomedical Engineering, Northwestern University（生物医学工程系，西北大学）；Dept. of Pediatrics, Northwestern University（儿科系，西北大学）
Markerless Multi-view 3D Human Pose Estimation: a survey
Instituto de Engenharia de Sistemas e Computadores, Tecnologia e Ciência (INESC TEC)（系统与计算机工程、技术与科学研究所（INESC TEC））；Faculdade de Engenharia da Universidade do Porto (FEUP)（波尔图大学工程学院（FEUP））；Faculdade de Ciências da Universidade do Porto (FCUP)（波尔图大学理学院（FCUP））
Shirley Ryan AbilityLab（雪莉·瑞安能力实验室）；Northwestern University（西北大学）
Markerless motion capture using computer vision and human pose estimation (HPE) has the potential to expand access to precise movement analysis. This could greatly benefit rehabilitation by enabling more accurate tracking of outcomes and providing more sensitive tools for research. There are numerous steps between obtaining videos to extracting accurate biomechanical results and limited research to guide many critical design decisions in these pipelines. In this work, we analyze several of these steps including the algorithm used to detect keypoints and the keypoint set, the approach to recons…
University of Freiburg（弗莱堡大学）；Adobe Research（Adobe研究院）
Estimating 3D hand pose from single RGB images is a highly ambiguous problem that relies on an unbiased training dataset. In this paper, we analyze cross-dataset generalization when training on existing datasets. We find that approaches perform well on the datasets they are trained on, but do not generalize to other datasets or in-the-wild scenarios. As a consequence, we introduce the first large-scale, multi-view hand dataset that is accompanied by both 3D hand pose and shape annotations. For annotating this real-world dataset, we propose an iterative, semi-automated `human-in-the-loop' appro…
We construct the first markerless deformable interaction dataset recording interactive motions of the hands and deformable objects, called HMDO (Hand Manipulation with Deformable Objects). With our built multi-view capture system, it captures the deformable interactions with multiple perspectives, various object shapes, and diverse interactive forms. Our motivation is the current lack of hand and deformable object interaction datasets, as 3D hand and deformable object reconstruction is challenging. Mainly due to mutual occlusion, the interaction area is difficult to observe, the visual feature…
Markerless motion capture has become an active field of research in computer vision in recent years. Its extensive applications are known in a great variety of fields, including computer animation, human motion analysis, biomedical research, virtual reality, and sports science. Estimating human posture has recently gained increasing attention in the computer vision community, but due to the depth of uncertainty and the lack of the synthetic datasets, it is a challenging task. Various approaches have recently been proposed to solve this problem, many of which are based on deep learning. They ar…
Simultaneous hyperkinetic movement disorders phenotyping: a cross-cohort pediatric transfer study using routine videos, markerless pose estimation and a tabular foundation model
Lausanne University Hospital (CHUV)（洛桑大学医院）；University of Lausanne (UNIL)（洛桑大学）；Institut du Neurone（神经元研究所）；Clinique Beau Soleil（博索莱伊诊所）；Institut Mutualiste Montpelliérain（蒙彼利埃互助研究所）；Military University Hospital of Sfax（斯法克斯军事大学医院）；University of Edinburgh（爱丁堡大学）；Hospital Sant Joan de Déu（圣琼德迪乌医院）；European Reference Network for Rare Neurological Diseases (ERN-RND)（欧洲罕见神经系统疾病参考网络）；Instituto de Salud Carlos III（卡洛斯三世健康研究所）；CHU Montpellier（蒙彼利埃大学医院）；Umeå University（于默奥大学）；University Hospital Lausanne（洛桑大学医院）；Ecole Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）；British Columbia Children’s Hospital（不列颠哥伦比亚儿童医院）
Biomechanics-aware Multi-view Markerless Motion Capture of Dexterous Hand Movements
Biomedical Engineering Department, Northwestern University（西北大学生物医学工程系）；Shirley Ryan AbilityLab（谢里尔·瑞安能力实验室）；Department of Organismal Biology and Anatomy, University of Chicago（芝加哥大学器官生物学与解剖学系）；Department of Physical Medicine and Rehabilitation, Northwestern University（西北大学物理医学与康复系）
KinTwin: Imitation Learning with Torque and Muscle Driven Biomechanical Models Enables Precise Replication of Able-Bodied and Impaired Movement from Markerless Motion Capture
Department of PM&R Northwestern University Shirley Ryan AbilityLab（运动医学系西北大学施里尔 Ryan 能力实验室）
Unmanned aerial vehicles (UAVs) have been used for many applications in recent years, from urban search and rescue, to agricultural surveying, to autonomous underground mine exploration. However, deploying UAVs in tight, indoor spaces, especially close to humans, remains a challenge. One solution, when limited payload is required, is to use micro-UAVs, which pose less risk to humans and typically cost less to replace after a crash. However, micro-UAVs can only carry a limited sensor suite, e.g. a monocular camera instead of a stereo pair or LiDAR, complicating tasks like dense mapping and mark…
NTT DOCOMO（日本电信电话多科莫公司）；The University of Tokyo（东京大学）
Although many studies have investigated markerless motion capture, the technology has not been applied to real sports or concerts. In this paper, we propose a markerless motion capture method with spatiotemporal accuracy and smoothness from multiple cameras in wide-space and multi-person environments. The proposed method predicts each person's 3D pose and determines the bounding box of multi-camera images small enough. This prediction and spatiotemporal filtering based on human skeletal model enables 3D reconstruction of the person and demonstrates high-accuracy. The accurate 3D reconstruction…
Phase-Separated Complex Hilbert PCA on Markerless 3D Pose Estimation Data: A Global Phase Network and Its Extension to a Continuous Field on the Body Surface
Faculty of Information Engineering, Kanazawa Gakuin University（信息工程学部，金泽神户大学）
Quantifying Rodda and Graham Gait Classification from 3D Markerless Kinematics derived from a Single-view Video in a Heterogeneous Pediatric Clinical Cohort
Department of Biomedical Informatics, Emory University（埃默里大学生物医学信息学系）；Shriners Children’s（夏皮罗儿童医院）；The Wallace H. Coulter Department of Biomedical Engineering, Emory University and Georgia Institute of Technology（埃默里大学和佐治亚理工学院的沃克·H·库勒生物医学工程系）
Uncertainty-Aware Mapping from 3D Keypoints to Anatomical Landmarks for Markerless Biomechanics
Department of Electrical and Information Engineering, University of Cassino and Southern Lazio（卡西诺和南拉齐奥大学电气与信息工程系）；Department of Research and Development, LUNEX（LUNEX研发部）
We present a novel data-driven regularizer for weakly-supervised learning of 3D human pose estimation that eliminates the drift problem that affects existing approaches. We do this by moving the stereo reconstruction problem into the loss of the network itself. This avoids the need to reconstruct 3D data prior to training and unlike previous semi-supervised approaches, avoids the need for a warm-up period of supervised training. The conceptual and implementational simplicity of our approach is fundamental to its appeal. Not only is it straightforward to augment many weakly-supervised approache…
Vicarious Perception Technologies (VicarVision)（Vicarious感知技术公司（VicarVision））；Delft University of Technology（代尔夫特理工大学）
Markerless estimation of 3D Kinematics has the great potential to clinically diagnose and monitor movement disorders without referrals to expensive motion capture labs; however, current approaches are limited by performing multiple de-coupled steps to estimate the kinematics of a person from videos. Most current techniques work in a multi-step approach by first detecting the pose of the body and then fitting a musculoskeletal model to the data for accurate kinematic estimation. Errors in training data of the pose detection algorithms, model scaling, as well the requirement of multiple cameras …
UC San Diego（加州大学圣迭戈分校）；Microsoft（微软）；University of Salzburg（萨尔茨堡大学）；Wormpex AI Research（Wormpex AI 研究院）
We propose a method to learn, even using a dataset where objects appear only in sparsely sampled views (e.g. Pix3D), the ability to synthesize a pose trajectory for an arbitrary reference image. This is achieved with a cross-modal pose trajectory transfer mechanism. First, a domain transfer function is trained to predict, from an RGB image of the object, its 2D depth map. Then, a set of image views is generated by learning to simulate object rotation in the depth space. Finally, the generated poses are mapped from this latent space into a set of corresponding RGB images using a learned identit…
Existing deep learning approaches on 3d human pose estimation for videos are either based on Recurrent or Convolutional Neural Networks (RNNs or CNNs). However, RNN-based frameworks can only tackle sequences with limited frames because sequential models are sensitive to bad frames and tend to drift over long sequences. Although existing CNN-based temporal frameworks attempt to address the sensitivity and drift problems by concurrently processing all input frames in the sequence, the existing state-of-the-art CNN-based framework is limited to 3d pose estimation of a single frame from a sequenti…
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；University of Stuttgart（斯图加特大学）
In this letter, we present a novel markerless 3D human motion capture (MoCap) system for unstructured, outdoor environments that uses a team of autonomous unmanned aerial vehicles (UAVs) with on-board RGB cameras and computation. Existing methods are limited by calibrated cameras and off-line processing. Thus, we present the first method (AirPose) to estimate human pose and shape using images captured by multiple extrinsically uncalibrated flying cameras. AirPose itself calibrates the cameras relative to the person instead of relying on any pre-calibration. It uses distributed neural networks …
Commonly used human motion capture systems require intrusive attachment of markers that are visually tracked with multiple cameras. In this work we present an efficient and inexpensive solution to markerless motion capture using only a few Kinect sensors. Unlike the previous work on 3d pose estimation using a single depth camera, we relax constraints on the camera location and do not assume a co-operative user. We apply recent image segmentation techniques to depth images and use curriculum learning to train our system on purely synthetic data. Our method accurately localizes body parts withou…
Hamburg University of Technology（汉堡工业大学）；eemagine Medical Imaging Solutions GmbH（eemagine医学影像解决方案有限公司）
Head pose estimation and tracking is useful in variety of medical applications. With the advent of RGBD cameras like Kinect, it has become feasible to do markerless tracking by estimating the head pose directly from the point clouds. One specific medical application is robot assisted transcranial magnetic stimulation (TMS) where any patient motion is compensated with the help of a robot. For increased patient comfort, it is important to track the head without markers. In this regard, we address the head pose estimation problem using two different approaches. In the first approach, we build upo…
We present BiPOCO, a Bi-directional trajectory predictor with POse COnstraints, for detecting anomalous activities of pedestrians in videos. In contrast to prior work based on feature reconstruction, our work identifies pedestrian anomalous events by forecasting their future trajectories and comparing the predictions with their expectations. We introduce a set of novel compositional pose-based losses with our predictor and leverage prediction errors of each body joint for pedestrian anomaly detection. Experimental results show that our BiPOCO approach can detect pedestrian anomalous activities…
Ferdowsi University of Mashhad（马什哈德菲尔多西大学）；University of Adelaide（阿德莱德大学）；Stanford University（斯坦福大学）；Monash University（莫纳什大学）
Joint forecasting of human trajectory and pose dynamics is a fundamental building block of various applications ranging from robotics and autonomous driving to surveillance systems. Predicting body dynamics requires capturing subtle information embedded in the humans' interactions with each other and with the objects present in the scene. In this paper, we propose a novel TRajectory and POse Dynamics (nicknamed TRiPOD) method based on graph attentional networks to model the human-human and human-object interactions both in the input space and the output space (decoded future output). The model…
Multi-person pose forecasting remains a challenging problem, especially in modeling fine-grained human body interaction in complex crowd scenarios. Existing methods typically represent the whole pose sequence as a temporal series, yet overlook interactive influences among people based on skeletal body parts. In this paper, we propose a novel Trajectory-Aware Body Interaction Transformer (TBIFormer) for multi-person pose forecasting via effectively modeling body part interactions. Specifically, we construct a Temporal Body Partition Module that transforms all the pose sequences into a Multi-Per…
Prediction with high accuracy is essential for various applications such as autonomous driving. Existing prediction models are easily prone to errors in real-world settings where observations (e.g. human poses and locations) are often noisy. To address this problem, we introduce GPRAR, a graph convolutional network based pose reconstruction and action recognition for human trajectory prediction. The key idea of GPRAR is to generate robust features: human poses and actions, under noisy scenarios. To this end, we design GPRAR using two novel sub-networks: PRAR (Pose Reconstruction and Action Rec…
