There has been significant progress in machine learning algorithms for human pose estimation that may provide immense value in rehabilitation and movement sciences. However, there remain several challenges to routine use of these tools for clinical practice and translational research, including: 1) high technical barrier to entry, 2) rapidly evolving space of algorithms, 3) challenging algorithmic interdependencies, and 4) complex data management requirements between these components. To mitigate these barriers, we developed a human pose estimation pipeline that facilitates running state-of-the-art algorithms on data acquired in clinical context. Our system allows for running different implementations of several classes of algorithms and handles their interdependencies easily. These algorithm classes include subject identification and tracking, 2D keypoint detection, 3D joint location estimation, and estimating the pose of body models. The system uses a database to manage videos, intermediate analyses, and data for computations at each stage. It also provides tools for data visualization, including generating video overlays that also obscure faces to enhance privacy. Our goal in this work is not to train new algorithms, but to advance the use of cutting-edge human pose estimation algorithms for clinical and translation research. We show that this tool facilitates analyzing large numbers of videos of human movement ranging from gait laboratories analyses, to clinic and therapy visits, to people in the community. We also highlight limitations of these algorithms when applied to clinical populations in a rehabilitation setting.
Monkey Transfer Learning Can Improve Human Pose Estimation
School of Medicine, Medical Sciences and Nutrition（医学、医学科学与营养学院）；University of Aberdeen（阿伯丁大学）；School of Health and Wellbeing（健康与福祉学院）；University of Glasgow（格拉斯哥大学）；School of Natural and Computing Sciences（自然与计算科学学院）；School of Engineering（工程学院）
University of California, San Diego（加利福尼亚大学圣地亚哥分校）
Deep learning has made a remarkable impact in the field of natural image processing over the past decade. Consequently, there is a great deal of interest in replicating this success across unsolved tasks in related domains, such as medical image analysis. Core to medical image analysis is the task of semantic segmentation which enables various clinical workflows. Due to the challenges inherent in manual segmentation, many decades of research have been devoted to discovering extensible, automated, expert-level segmentation techniques. Given the groundbreaking performance demonstrated by recent …
3D object detection and pose estimation has been studied extensively in recent decades for its potential applications in robotics. However, there still remains challenges when we aim at detecting multiple objects while retaining low false positive rate in cluttered environments. This paper proposes a robust 3D object detection and pose estimation pipeline based on RGB-D images, which can detect multiple objects simultaneously while reducing false positives. Detection begins with template matching and yields a set of template matches. A clustering algorithm then groups templates of similar spat…
Shanghai University（上海大学）；Shanghai Institute of Advanced Communication and Data Science（上海先进通信与数据科学研究院）；JD Logistics（京东物流）；JD Explore Academy（京东探索研究院）
Human pose estimation aims to figure out the keypoints of all people in different scenes. Current approaches still face some challenges despite promising results. Existing top-down methods deal with a single person individually, without the interaction between different people and the scene they are situated in. Consequently, the performance of human detection degrades when serious occlusion happens. On the other hand, existing bottom-up methods consider all people at the same time and capture the global knowledge of the entire image. However, they are less accurate than the top-down methods d…
Department of Mechanical and Process Engineering（机械与过程工程系）
We propose a complete pipeline that allows object detection and simultaneously estimate the pose of these multiple object instances using just a single image. A novel "keypoint regression" scheme with a cross-ratio term is introduced that exploits prior information about the object's shape and size to regress and find specific feature points. Further, a priori 3D information about the object is used to match 2D-3D correspondences and accurately estimate object positions up to a distance of 15m. A detailed discussion of the results and an in-depth analysis of the pipeline is presented. The pipe…
Universiti Teknologi Brunei（文莱理工大学）；National Institute of Information and Communications Technology（日本国立信息通信研究机构）；National Electronics and Computer Technology Center (NECTEC)（泰国国家电子与计算机技术中心）；CLEALINKTECHNOLOGY Co., Ltd.（CLEALINK科技有限公司）
Recent advances in deep learning and computer vision offer an excellent opportunity to investigate high-level visual analysis tasks such as human localization and human pose estimation. Although the performance of human localization and human pose estimation has significantly improved in recent reports, they are not perfect and erroneous localization and pose estimation can be expected among video frames. Studies on the integration of these techniques into a generic pipeline that is robust to noise introduced from those errors are still lacking. This paper fills the missing study. We explored …
We propose a new method for estimating the relative pose between two images, where we jointly learn keypoint detection, description extraction, matching and robust pose estimation. While our architecture follows the traditional pipeline for pose estimation from geometric computer vision, all steps are learnt in an end-to-end fashion, including feature matching. We demonstrate our method for the task of visual localization of a query image within a database of images with known pose. Pairwise pose estimation has many practical applications for robotic mapping, navigation, and AR. For example, t…
Pose as Clinical Prior: Learning Dual Representations for Scoliosis Screening
Southern University of Science and Technology（南方科技大学）；The Hong Kong Polytechnic University（香港理工大学）；School of Artificial Intelligence, Shenzhen University（深圳大学人工智能学院）
Insight: A Multi-Modal Diagnostic Pipeline using LLMs for Ocular Surface Disease Diagnosis
Clinical Research Center, University of California, Berkeley（加州大学伯克利分校临床研究中心）；University of California, Berkeley（加州大学伯克利分校）；California Institute of Technology（加州理工学院）；University of California, Davis（加州大学戴维斯分校）；University of Hong Kong（香港大学）
CAD-RADS scoring of coronary CT angiography with Multi-Axis Vision Transformer: a clinically-inspired deep learning pipeline
University of Pavia（帕维亚大学）；Centro Cardiologico Monzino IRCCS（蒙齐诺心脏病学中心IRCCS）；University of Milan（米兰大学）；IRCCS Istituti Clinici Scientifici Maugeri（IRCCS莫杰里科学临床研究所）
Equivariant symmetry-aware head pose estimation for fetal MRI
MIT Computer Science and Artificial Intelligence Laboratory（麻省理工学院计算机科学与人工智能实验室）；Boston Children's Hospital and Harvard Medical School（波士顿儿童医院和哈佛医学院）；Inria, Université Côte d'Azur（法国国家科学研究中心（Inria）、蔚蓝海岸大学）
Carnegie Mellon University（卡内基梅隆大学）；University of California, Berkeley（加州大学伯克利分校）
Many autonomous systems forecast aspects of the future in order to aid decision-making. For example, self-driving vehicles and robotic manipulation systems often forecast future object poses by first detecting and tracking objects. However, this detect-then-forecast pipeline is expensive to scale, as pose forecasting algorithms typically require labeled sequences of object poses, which are costly to obtain in 3D space. Can we scale performance without requiring additional labels? We hypothesize yes, and propose inverting the detect-then-forecast pipeline. Instead of detecting, tracking and the…
Video-Based Markerless Motion Capture for Clinical and Rehabilitation Biomechanics: A PRISMA-ScR Scoping Review of Validated Architectures, Clinical Readiness, and Emerging Methods
Université Côte d'Azur（蔚蓝海岸大学）；Centre Hospitalier Universitaire de Nice（尼斯大学附属医院）；Institut Universitaire de France（法国大学研究院）
A Detection-Gated Pipeline for Robust Glottal Area Waveform Extraction and Clinical Pathology Assessment
Orchard Robotics；Department of Speech, Language, and Hearing Sciences（语音、语言和听力科学系）；Indiana University（印第安纳大学）；Department of Otolaryngology–Head and Neck Surgery（耳鼻喉科-头颈外科系）；Indiana University School of Medicine（印第安纳大学医学院）
A Multimodal Pipeline for Clinical Data Extraction: Applying Vision-Language Models to Scans of Transfusion Reaction Reports
Institute for Transfusion Medicine, University Hospital Essen, Hufelandstraße 55, Essen, Germany（1 输血医学研究所，埃森大学医院，Hufelandstraße 55，德国）；Department of Computer Science, University of Applied Sciences and Arts Dortmund (FHDO), Emil-Figge Str. 42, Dortmund, Germany（2 计算机科学系，多特蒙德应用科学大学（FHDO），Emil-Figge Str. 42，德国）；Institute for Medical Informatics, Biometry and Epidemiology (IMIBE), University Hospital Essen, Hufelandstraße 55, Essen, Germany（3 医学信息学、生物统计学和流行病学研究所（IMIBE），埃森大学医院，Hufelandstraße 55，德国）；Institute for AI in Medicine (IKIM), University Hospital Essen, Girardetstraße 2, Essen, Germany（4 医学人工智能研究所（IKIM），埃森大学医院，Girardetstraße 2，德国）；Institute of Interventional and Diagnostic Radiology and Neuroradiology, University Hospital Essen, Hufelandstraße 55, Essen, Germany（5 干预性和诊断放射学及神经放射学研究所，埃森大学医院，Hufelandstraße 55，德国）；Department of Dermatology, University Hospital Essen, Hufelandstraße 55, Essen, Germany（6 皮肤科系，埃森大学医院，Hufelandstraße 55，德国）
AnatomicalNets: A Multi-Structure Segmentation and Contour-Based Distance Estimation Pipeline for Clinically Grounded Lung Cancer T-Staging
Department of Robotics and Mechatronics Engineering, University of Dhaka（达卡大学机器人与机电工程系）；Department of Electrical and Electronics Engineering, University of Dhaka（达卡大学电气与电子工程系）；Department of Electrical Engineering, Qatar University（卡塔尔大学电气工程系）；Department of Radiology, Hamad Medical Corporation（哈马德医疗集团放射科）；Department of Biomedical Technology, Prince Sattam Bin Abdulaziz University（普林斯·萨塔姆·本·阿卜杜勒阿齐兹大学生物医学技术系）
Stroke Lesion Segmentation in Clinical Workflows: A Modular, Lightweight, and Deployment-Ready Tool
Univ Rennes（里昂大学）；Inria（法国国家信息与自动化技术研究所）；CNRS（法国国家科学研究中心）；Inserm（法国国家健康与医学研究院）；IRISA UMR 6074；Siemens Healthineers（西门子医疗）；CHU Rennes（里昂大学医院）；Centre de Kerpape（凯帕佩中心）
Camera pose estimation in large-scale environments is still an open question and, despite recent promising results, it may still fail in some situations. The research so far has focused on improving subcomponents of estimation pipelines, to achieve more accurate poses. However, there is no guarantee for the result to be correct, even though the correctness of pose estimation is critically important in several visual localization applications,such as in autonomous navigation. In this paper we bring to attention a novel research question, pose confidence estimation,where we aim at quantifying ho…
PoseDreamer: Scalable and Photorealistic Human Data Generation Pipeline with Diffusion Models
The Podium Institute for Sports Medicine and Technology, University of Oxford（牛津大学体育医学与技术讲台研究所）；Visual Geometry Group, University of Oxford（牛津大学视觉几何组）；Ukrainian Catholic University（乌克兰天主教大学）
Vanderbilt University（范德堡大学）；Vanderbilt University Medical Center（范德堡大学医学中心）；U.S. Department of Veterans Affairs（美国退伍军人事务部）
Image Quality Assessment (IQA) is important for scientific inquiry, especially in medical imaging and machine learning. Potential data quality issues can be exacerbated when human-based workflows use limited views of the data that may obscure digital artifacts. In practice, multiple factors such as network issues, accelerated acquisitions, motion artifacts, and imaging protocol design can impede the interpretation of image collections. The medical image processing community has developed a wide variety of tools for the inspection and validation of imaging data. Yet, IQA of computed tomography …
In this work, we propose a method that combines unsupervised deep learning predictions for optical flow and monocular disparity with a model based optimization procedure for instantaneous camera pose. Given the flow and disparity predictions from the network, we apply a RANSAC outlier rejection scheme to find an inlier set of flows and disparities, which we use to solve for the relative camera pose in a least squares fashion. We show that this pipeline is fully differentiable, allowing us to combine the pose with the network outputs as an additional unsupervised training loss to further refine…
Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究所）；Deloitte Analytics Institute（德勤分析研究所）；Moscow Research and Clinical Center for Neuropsychiatry（莫斯科神经精神病学研究和临床中心）
We consider a problem of diagnostic pattern recognition/classification from neuroimaging data. We propose a common data analysis pipeline for neuroimaging-based diagnostic classification problems using various ML algorithms and processing toolboxes for brain imaging. We illustrate the pipeline application by discovering new biomarkers for diagnostics of epilepsy and depression based on clinical and MRI/fMRI data for patients and healthy volunteers.
Peking University（北京大学）；Stanford University（斯坦福大学）；Shandong University（山东大学）；UCSD（加州大学圣地亚哥分校）；Tencent AI Lab（腾讯AI实验室）；Beijing Institute for General AI（北京通用人工智能研究院）
In this work, we tackle the problem of category-level online pose tracking of objects from point cloud sequences. For the first time, we propose a unified framework that can handle 9DoF pose tracking for novel rigid object instances as well as per-part pose tracking for articulated objects from known categories. Here the 9DoF pose, comprising 6D pose and 3D size, is equivalent to a 3D amodal bounding box representation with free 6D pose. Given the depth point cloud at the current frame and the estimated pose from the last frame, our novel end-to-end pipeline learns to accurately update the pos…
Automated Perceptually-Motivated Assessment of Photographic Consistency in Paired Clinical Photographs: Pipeline Development and Internal Evaluation
David Geffen School of Medicine at UCLA（加州大学洛杉矶分校大卫·格芬医学院）；Cornell University（康奈尔大学）；Tulane University（杜兰大学）；University of Pittsburgh School of Medicine（匹兹堡大学医学院）
Multi-modal 3D Pose and Shape Estimation with Computed Tomography
School of Computer Science, The University of Sydney, Sydney, NSW 2006, Australia（悉尼大学计算机科学学院）；School of Biomedical Engineering, The University of Sydney, Sydney, NSW 2006, Australia（悉尼大学生物医学工程学院）；Trauma Service, Westmead Hospital, Australia（西墨迪医院创伤服务）
Technical University of Munich（慕尼黑工业大学）；Siemens AG（西门子公司）
We propose a lightweight retrieval-based pipeline to predict 6DOF camera poses from RGB images. Our pipeline uses a convolutional neural network (CNN) to encode a query image as a feature vector. A nearest neighbor lookup finds the pose-wise nearest database image. A siamese convolutional neural network regresses the relative pose from the nearest neighboring database image to the query image. The relative pose is then applied to the nearest neighboring absolute pose to obtain the query image's final absolute pose prediction. Our model is a distilled version of NN-Net that reduces its paramete…
Ubisoft China AI & Data Lab（育碧中国AI与数据实验室）；Sichuan University（四川大学）；Ubisoft, Singapore（育碧新加坡）；INRIA Nancy Grand Est（法国国家信息与自动化研究所南锡大东区中心）
This work introduces a novel strategy for generating synthetic training data for 2D and 3D pose estimation of animals using keyframe animations. With the objective to automate the process of creating animations for wildlife, we train several 2D and 3D pose estimation models with synthetic data, and put in place an end-to-end pipeline called ZooBuilder. The pipeline takes as input a video of an animal in the wild, and generates the corresponding 2D and 3D coordinates for each joint of the animal's skeleton. With this approach, we produce motion capture data that can be used to create animations…
Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）
The Histogram of Oriented Gradient (HOG) descriptor has led to many advances in computer vision over the last decade and is still part of many state of the art approaches. We realize that the associated feature computation is piecewise differentiable and therefore many pipelines which build on HOG can be made differentiable. This lends to advanced introspection as well as opportunities for end-to-end optimization. We present our implementation of $\nabla$HOG based on the auto-differentiation toolbox Chumpy and show applications to pre-image visualization and pose estimation which extends the e…
Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）；Erasmus MC - University Medical Center Rotterdam（伊拉斯谟医学中心——鹿特丹大学医学中心）；University of Copenhagen（哥本哈根大学）；Massachusetts Institute of Technology（麻省理工学院）
Registration is a core component of many imaging pipelines. In case of clinical scans, with lower resolution and sometimes substantial motion artifacts, registration can produce poor results. Visual assessment of registration quality in large clinical datasets is inefficient. In this work, we propose to automatically assess the quality of registration to an atlas in clinical FLAIR MRI scans of the brain. The method consists of automatically segmenting the ventricles of a given scan using a neural network, and comparing the segmentation to the atlas' ventricles propagated to image space. We use…
STROKEVISION-BENCH: A Multimodal Video And 2D Pose Benchmark For Tracking Stroke Recovery
Center for Research in Computer Vision, University of Central Florida（计算机视觉研究中心，佛罗里达中央大学）；Mechanical and Aerospace Engineering, University of Central Florida（机械与航空航天工程，佛罗里达中央大学）
The conventional pose estimation of a 3D object usually requires the knowledge of the 3D model of the object. Even with the recent development in convolutional neural networks (CNNs), a 3D model is often necessary in the final estimation. In this paper, we propose a two-stage pipeline that takes in raw colored point cloud data and estimates an object's translation and rotation by running 3D convolutions on voxels. The pipeline is simple yet highly accurate: translation error is reduced to the voxel resolution (around 1 cm) and rotation error is around 5 degrees. The pipeline is also put to act…
Optimizing Stroke Risk Prediction: A Machine Learning Pipeline Combining ROS-Balanced Ensembles and XAI
1 Department of Robotics, Robo Tech Valley, Dhaka, Bangladesh 2 Department of Computer Science；Engineering, Jagannath University, Dhaka, Bangladesh 3 Department of Internet of Things \& Robotics Engineering, University of Frontier Technology, Bangladesh
In this paper, we propose a novel 3D graph convolution based pipeline for category-level 6D pose and size estimation from monocular RGB-D images. The proposed method leverages an efficient 3D data augmentation and a novel vector-based decoupled rotation representation. Specifically, we first design an orientation-aware autoencoder with 3D graph convolution for latent feature learning. The learned latent feature is insensitive to point shift and size thanks to the shift and scale-invariance properties of the 3D graph convolution. Then, to efficiently decode the rotation information from the lat…
Unlocking Multi-Site Clinical Data: A Federated Approach to Privacy-First Child Autism Behavior Analysis
Institute of Artificial Intelligence, University of Central Florida（中佛罗里达大学人工智能研究所）；Department of Clinical Sciences, College of Medicine, University of Central Florida（中佛罗里达大学医学院临床科学系）；Department of Computer Science, University of North Carolina at Charlotte（北卡罗来纳大学夏洛特分校计算机科学系）；Department of Computer Science, Northern Illinois University（北伊利诺伊大学计算机科学系）；Industrial and Systems Engineering, Northern Illinois University（北伊利诺伊大学工业与系统工程系）
