Semi-supervised Body Parsing and Pose Estimation for Enhancing Infant General Movement Assessment
The Pennsylvania State University（宾夕法尼亚州立大学）；Johns Hopkins University（约翰斯·霍普金斯大学）；Shenzhen Baoan Women’s and Children’s Hospital, Jinan University（暨南大学深圳宝安妇幼保健院）；Xi’an University of Architecture and Technology（西安建筑科技大学）
General movement assessment (GMA) of infant movement videos (IMVs) is an effective method for early detection of cerebral palsy (CP) in infants. We demonstrate in this paper that end-to-end trainable neural networks for image sequence recognition can be applied to achieve good results in GMA, and more importantly, augmenting raw video with infant body parsing and pose estimation information can significantly improve performance. To solve the problem of efficiently utilizing partially labeled IMVs for body parsing, we propose a semi-supervised model, termed SiamParseNet (SPN), which consists of two branches, one for intra-frame body parts segmentation and another for inter-frame label propagation. During training, the two branches are jointly trained by alternating between using input pairs of only labeled frames and input of both labeled and unlabeled frames. We also investigate training data augmentation by proposing a factorized video generative adversarial network (FVGAN) to synthesize novel labeled frames for training. When testing, we employ a multi-source inference mechanism, where the final result for a test frame is either obtained via the segmentation branch or via propagation from a nearby key frame. We conduct extensive experiments for body parsing using SPN on two infant movement video datasets, where SPN coupled with FVGAN achieves state-of-the-art performance. We further demonstrate that SPN can be easily adapted to the infant pose estimation task with superior performance. Last but not least, we explore the clinical application of our method for GMA. We collected a new clinical IMV dataset with GMA annotations, and our experiments show that SPN models for body parsing and pose estimation trained on the first two datasets generalize well to the new clinical dataset and their results can significantly boost the CRNN-based GMA prediction performance.
The Pennsylvania State University（宾夕法尼亚州立大学）；Xi’an University of Architecture and Technology（西安建筑科技大学）
General movement assessment (GMA) of infant movement videos (IMVs) is an effective method for the early detection of cerebral palsy (CP) in infants. Automated body parsing is a crucial step towards computer-aided GMA, in which infant body parts are segmented and tracked over time for movement analysis. However, acquiring fully annotated data for video-based body parsing is particularly expensive due to the large number of frames in IMVs. In this paper, we propose a semi-supervised body parsing model, termed SiamParseNet (SPN), to jointly learn single frame body parsing and label propagation be…
Aalto University（阿尔托大学）；Helsinki University Hospital（赫尔辛基大学医院）；University of Helsinki（赫尔辛基大学）；Tampere University（坦佩雷大学）；Scuola Superiore San’Anna（圣安娜高等学校）；IRCCS Fondazione Stella Maris（斯特拉·马里斯IRCCS基金会）；Kaasa solution GmbH（卡阿萨解决方案有限公司）
Infants' spontaneous and voluntary movements mirror developmental integrity of brain networks since they require coordinated activation of multiple sites in the central nervous system. Accordingly, early detection of infants with atypical motor development holds promise for recognizing those infants who are at risk for a wide range of neurodevelopmental disorders (e.g., cerebral palsy, autism spectrum disorders). Previously, novel wearable technology has shown promise for offering efficient, scalable and automated methods for movement assessment in adults. Here, we describe the development of …
Norwegian University of Science and Technology（挪威科技大学）；St. Olavs Hospital, Trondheim University Hospital（圣奥拉夫医院，特隆赫姆大学医院）
Assessment of spontaneous movements can predict the long-term developmental disorders in high-risk infants. In order to develop algorithms for automated prediction of later disorders, highly precise localization of segments and joints by infant pose estimation is required. Four types of convolutional neural networks were trained and evaluated on a novel infant pose dataset, covering the large variation in 1 424 videos from a clinical international community. The localization performance of the networks was evaluated as the deviation between the estimated keypoint positions and human expert ann…
University of Cambridge（剑桥大学）；University of Glasgow（格拉斯哥大学）
Infants' neurological development is heavily influenced by their motor skills. Evaluating a baby's movements is key to understanding possible risks of developmental disorders in their growth. Previous research in psychology has shown that measuring specific movements or gestures such as face touches in babies is essential to analyse how babies understand themselves and their context. This research proposes the first automatic approach that detects face touches from video recordings by tracking infants' movements and gestures. The study uses a multimodal feature fusion approach mixing spatial a…
Hiroshima University（广岛大学）；Prefectural University of Hiroshima（广岛县立大学）；University of Genoa（热那亚大学）；Istituto G.Gaslini Pediatric Hospital（G.加斯利尼儿科研究所）；Italian Institute of Technology（意大利技术研究院）
The assessment of general movements (GMs) in infants is a useful tool in the early diagnosis of neurodevelopmental disorders. However, its evaluation in clinical practice relies on visual inspection by experts, and an automated solution is eagerly awaited. Recently, video-based GMs classification has attracted attention, but this approach would be strongly affected by irrelevant information, such as background clutter in the video. Furthermore, for reliability, it is necessary to properly extract the spatiotemporal features of infants during GMs. In this study, we propose an automated GMs clas…
Fraunhofer Institute of Optronics, System Technologies and Image Exploitation（弗劳恩霍夫光电、系统技术与图像开发研究所）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；University Medical Center Freiburg（弗莱堡大学医学中心）；University of Freiburg（弗莱堡大学）；Ludwig Maximilian University（路德维希-马克西米利安大学）；Hauner Children’s Hospital（豪纳儿童医院）
Statistical models of the human body surface are generally learned from thousands of high-quality 3D scans in predefined poses to cover the wide variety of human body shapes and articulations. Acquisition of such data requires expensive equipment, calibration procedures, and is limited to cooperative subjects who can understand and follow instructions, such as adults. We present a method for learning a statistical 3D Skinned Multi-Infant Linear body model (SMIL) from incomplete, low-quality RGB-D sequences of freely moving infants. Quantitative experiments show that SMIL faithfully represents …
Cerebral Palsy Alliance Research Institute（脑瘫联盟研究所）；The University of Sydney（悉尼大学）；The Children’s Hospital at Westmead（韦斯特米德儿童医院）
The absence or abnormality of fidgety movements of joints or limbs is strongly indicative of cerebral palsy in infants. Developing computer-based methods for assessing infant movements in videos is pivotal for improved cerebral palsy screening. Most existing methods use appearance-based features and are thus sensitive to strong but irrelevant signals caused by background clutter or a moving camera. Moreover, these features are computed over the whole frame, thus they measure gross whole body movements rather than specific joint/limb motion. Addressing these challenges, we develop and validate …
Northumbria University（诺森比亚大学）；NUTH NHS Foundation Trust（NUTH NHS基金会信托）
Providing early diagnosis of cerebral palsy (CP) is key to enhancing the developmental outcomes for those affected. Diagnostic tools such as the General Movements Assessment (GMA), have produced promising results in early diagnosis, however these manual methods can be laborious. In this paper, we propose a new framework for the automated classification of infant body movements, based upon the GMA, which unlike previous methods, also incorporates a visualization framework to aid with interpretability. Our proposed framework segments extracted features to detect the presence of Fidgety Movements…
Universitá Politecnica delle Marche（马尔凯理工大学）；Istituto Italiano di Tencologia（意大利技术研究院）
Preterm infants' limb-pose estimation is a crucial but challenging task, which may improve patients' care and facilitate clinicians in infant's movements monitoring. Work in the literature either provides approaches to whole-body segmentation and tracking, which, however, has poor clinical value, or retrieve a posteriori limb pose from limb segmentation, increasing computational costs and introducing inaccuracy sources. In this paper, we address the problem of limb-pose estimation under a different point of view. We proposed a 2D fully-convolutional neural network for roughly detecting limb jo…
To make the earlier medical intervention of infants' cerebral palsy (CP), early diagnosis of brain damage is critical. Although general movements assessment(GMA) has shown promising results in early CP detection, it is laborious. Most existing works take videos as input to make fidgety movements(FMs) classification for the GMA automation. Those methods require a complete observation of videos and can not localize video frames containing normal FMs. Therefore we propose a novel approach named WO-GMA to perform FMs localization in the weakly supervised online setting. Infant body keypoints are f…
Markerless Motion Capture for Biomechanical Whole-Body Kinematic Estimation in Infants
Center for Bionic Medicine, Shirley Ryan AbilityLab（生物医学中心，Shirley Ryan AbilityLab）；Dept. of Physical Therapy and Human Movement Science, Northwestern University（物理治疗与人类运动科学系，西北大学）；Dept. of Biomedical Engineering, Northwestern University（生物医学工程系，西北大学）；Dept. of Pediatrics, Northwestern University（儿科系，西北大学）
Durham University（杜伦大学）；University of Glasgow（格拉斯哥大学）
Early prediction is clinically considered one of the essential parts of cerebral palsy (CP) treatment. We propose to implement a low-cost and interpretable classification system for supporting CP prediction based on General Movement Assessment (GMA). We design a Pytorch-based attention-informed graph convolutional network to early identify infants at risk of CP from skeletal data extracted from RGB videos. We also design a frequency-binning module for learning the CP movements in the frequency domain while filtering noise. Our system only requires consumer-grade RGB videos for training to supp…
Carleton University（卡尔顿大学）；Children’s Hospital of Eastern Ontario（东安大略儿童医院）
Infant pose monitoring during sleep has multiple applications in both healthcare and home settings. In a healthcare setting, pose detection can be used for region of interest detection and movement detection for noncontact based monitoring systems. In a home setting, pose detection can be used to detect sleep positions which has shown to have a strong influence on multiple health factors. However, pose monitoring during sleep is challenging due to heavy occlusions from blanket coverings and low lighting. To address this, we present a novel dataset, Simultaneously-collected multimodal Mannequin…
Aalto University（阿尔托大学）；Helsinki University Hospital（赫尔辛基大学医院）；University of Helsinki（赫尔辛基大学）；Tampere University（坦佩雷大学）
Infant motility assessment using intelligent wearables is a promising new approach for assessment of infant neurophysiological development, and where efficient signal analysis plays a central role. This study investigates the use of different end-to-end neural network architectures for processing infant motility data from wearable sensors. We focus on the performance and computational burden of alternative sensor encoder and time-series modelling modules and their combinations. In addition, we explore the benefits of data augmentation methods in ideal and non-ideal recording conditions. The ex…
Object segmentation in infant's egocentric videos is a fundamental step in studying how children perceive objects in early stages of development. From the computer vision perspective, object segmentation in such videos pose quite a few challenges because the child's view is unfocused, often with large head movements, effecting in sudden changes in the child's point of view which leads to frequent change in object properties such as size, shape and illumination. In this paper, we develop a semi-automated, domain specific, method to address these concerns and facilitate the object annotation pro…
Comparison of marker-less 2D image-based methods for infant pose estimation
University Medical Center Göttingen（戈滕堡大学医学中心）；German Center for Child and Adolescent Health (DZKJ)（德国儿童与青少年健康研究中心）；Leibniz ScienceCampus Göttingen（戈滕堡莱比锡科学校园）；University of Göttingen（戈滕堡大学）；III Institute of Physics - Biophysics（物理-生物物理第三研究所）；University Hospital Heidelberg（海德堡大学医院）；Ruprecht-Karls University of Heidelberg（海德堡鲁珀特-卡尔大学）；Department of Child and Adolescent Psychiatry（儿童与青少年精神病科部门）；Medical University of Graz（格拉茨医学院）；Curtin Autism Research Group（库林自闭症研究组）；Curtin School of Allied Health（库林健康联盟学校）；Curtin University（库林大学）；Center of Neurodevelopmental Disorders (KIND)（神经发育障碍中心（KIND））；Department of Women’s and Children’s Health（妇女与儿童健康部门）；Center for Psychiatry Research（精神医学研究中心）；Karolinska Institutet（卡罗林斯卡研究所）；Region Stockholm（斯德哥尔摩地区）；Child and Adolescent Psychiatry, Stockholm Health Care Services, Region Stockholm（斯德哥尔摩健康服务儿童与青少年精神病科，斯德哥尔摩地区）；Stockholm Health Care Services, Region Stockholm（斯德哥尔摩健康服务，斯德哥尔摩地区）
Shenzhen Automatic Rehabilitation Laboratory（深圳市自动康复实验室）；New York University（纽约大学）；Shenzhen Children’s Hospital（深圳市儿童医院）
Movement and pose assessment of newborns lets experienced pediatricians predict neurodevelopmental disorders, allowing early intervention for related diseases. However, most of the newest AI approaches for human pose estimation methods focus on adults, lacking publicly benchmark for infant pose estimation. In this paper, we fill this gap by proposing infant pose dataset and Deep Aggregation Vision Transformer for human pose estimation, which introduces a fast trained full transformer framework without using convolution operations to extract features in the early stages. It generalizes Transfor…
Towards Scalable Newborn Screening: Automated General Movement Assessment in Uncontrolled Settings
ETH Zurich（苏黎世联邦理工学院）；University of Zurich（苏黎世大学）；University of Regensburg（莱茵河畔埃尔兰根大学）；University Children’s Hospital（儿童医院）；University Children’s Hospital Regensburg（埃尔兰根儿童医院）
University Medical Center Göttingen（哥廷根大学医学中心）；Karolinska Institutet（卡罗林斯卡学院）；Medical University of Graz（格拉茨医科大学）；Leibniz-ScienceCampus Primate Cognition（莱布尼茨灵长类认知科学园区）；David Geffen UCLA School of Medicine（加州大学洛杉矶分校大卫·格芬医学院）；Region Stockholm（斯德哥尔摩地区）；Curtin University（科廷大学）；Victoria University of Wellington（惠灵顿维多利亚大学）
Video recording is a widely used method for documenting infant and child behaviours in research and clinical practice. Video data has rarely been shared due to ethical concerns of confidentiality, although the need of shared large-scaled datasets remains increasing. This demand is even more imperative when data-driven computer-based approaches are involved, such as screening tools to complement clinical assessments. To share data while abiding by privacy protection rules, a critical question arises whether efforts at data de-identification reduce data utility? We addressed this question by sho…
Imperial College London（伦敦帝国理工学院）；Evelina London Children’s Hospital（伦敦伊芙莉娜儿童医院）；King’s College London（伦敦国王学院）
Human pose estimation is a major computer vision problem with applications ranging from augmented reality and video capture to surveillance and movement tracking. In the medical context, the latter may be an important biomarker for neurological impairments in infants. Whilst many methods exist, their application has been limited by the need for well annotated large datasets and the inability to generalize to humans of different shapes and body compositions, e.g. children and infants. In this paper we present a novel method for learning pose estimators for human adults and infants in an unsuper…
Action parsing in videos with complex scenes is an interesting but challenging task in computer vision. In this paper, we propose a generic 3D convolutional neural network in a multi-task learning manner for effective Deep Action Parsing (DAP3D-Net) in videos. Particularly, in the training phase, action localization, classification and attributes learning can be jointly optimized on our appearancemotion data via DAP3D-Net. For an upcoming test video, we can describe each individual action in the video simultaneously as: Where the action occurs, What the action is and How the action is performe…
