The raising availability of 3D cameras and dramatic improvement of computer vision algorithms in the recent decade, accelerated the research of automatic movement assessment solutions. Such solutions can be implemented at home, using affordable equipment and dedicated software. In this paper, we divide the movement assessment task into secondary tasks and explain why they are needed and how they can be addressed. We review the recent solutions for automatic movement assessment from skeleton videos, comparing them by their objectives, features, movement domains and algorithmic approaches. In addition, we discuss the status of the research on this topic in a high level.
The task of assessing movement quality has recently gained high demand in a variety of domains. The ability to automatically assess subject movement in videos that were captured by affordable devices, such as Kinect cameras, is essential for monitoring clinical rehabilitation processes, for improving motor skills and for movement learning tasks. The need to pay attention to low-level details while accurately tracking the movement stages, makes this task very challenging. In this work, we introduce A-MAL, an automatic, strong movement assessment learning algorithm that only learns from properly…
A Study on the Matching Rate of Dance Movements Using 2D Skeleton Detection and 3D Pose Estimation: Why Is SEVENTEEN's Performance So Bita-Zoroi (Perfectly Synchronized)?
Skeletal Movement to Color Map: A Novel Representation for 3D Action Recognition with Inception Residual Networks
Carlos III University of Madrid（马德里卡洛斯三世大学）；Cerema Research Center（Cerema研究中心）；Institut de Recherche en Informatique de Toulouse（图卢兹计算机科学研究所）；Université de Toulouse（图卢兹大学）
University of Bristol（布里斯托大学）；University of Toulon（土伦大学）
We propose a view-invariant method towards the assessment of the quality of human movements which does not rely on skeleton data. Our end-to-end convolutional neural network consists of two stages, where at first a view-invariant trajectory descriptor for each body joint is generated from RGB images, and then the collection of trajectories for all joints are processed by an adapted, pre-trained 2D CNN (e.g. VGG-19 or ResNeXt-50) to learn the relationship amongst the different body parts and deliver a score for the movement quality. We release the only publicly-available, multi-view, non-skelet…
Key Laboratory of Machine Perception, Shenzhen Graduate School, Peking University（北京大学深圳研究生院机器感知重点实验室）；The School of Intelligent Systems Engineering, Sun Yat-sen University（中山大学智能系统工程学院）
In recent years, self-supervised representation learning for skeleton-based action recognition has been developed with the advance of contrastive learning methods. The existing contrastive learning methods use normal augmentations to construct similar positive samples, which limits the ability to explore novel movement patterns. In this paper, to make better use of the movement patterns introduced by extreme augmentations, a Contrastive Learning framework utilizing Abundant Information Mining for self-supervised action Representation (AimCLR) is proposed. First, the extreme augmentations and t…
In this paper, we propose a novel approach to convert given speech audio to a photo-realistic speaking video of a specific person, where the output video has synchronized, realistic, and expressive rich body dynamics. We achieve this by first generating 3D skeleton movements from the audio sequence using a recurrent neural network (RNN), and then synthesizing the output video via a conditional generative adversarial network (GAN). To make the skeleton movement realistic and expressive, we embed the knowledge of an articulated 3D human skeleton and a learned dictionary of personal speech iconic…
Read My Ears! Horse Ear Movement Detection for Equine Affective State Assessment
Visual Analysis and Perception Lab, Aalborg University（奥尔堡大学视觉分析与感知实验室）；Department of Veterinary Clinical Sciences, University of Copenhagen（哥本哈根大学兽医临床科学系）
University of Bristol（布里斯托大学）；Southmead Hospital（南米德医院）
Evaluating neurological disorders such as Parkinson's disease (PD) is a challenging task that requires the assessment of several motor and non-motor functions. In this paper, we present an end-to-end deep learning framework to measure PD severity in two important components, hand movement and gait, of the Unified Parkinson's Disease Rating Scale (UPDRS). Our method leverages on an Inflated 3D CNN trained by a temporal segment framework to learn spatial and long temporal structure in video data. We also deploy a temporal attention mechanism to boost the performance of our model. Further, motion…
Towards Objective Obstetric Ultrasound Assessment: Contrastive Representation Learning for Fetal Movement Detection
Department of Electrical and Computer Systems Engineering, Monash University（莫纳什大学电气与计算机系统工程系）；AIM for Health Lab, Faculty of Information Technology, Monash University（健康AIM实验室，信息技术学院，莫纳什大学）；School of Biomedical Engineering, University of Sydney（悉尼大学生物医学工程学院）；Department of Chemical & Biological Engineering, Monash University（莫纳什大学化学与生物工程系）；Department of Obstetrics & Gynaecology, Monash University（莫纳什大学妇产科系）
Indian Institute of Technology Madras（印度理工学院马德拉斯分校）
Physical rehabilitation programs frequently begin with a brief stay in the hospital and continue with home-based rehabilitation. Lack of feedback on exercise correctness is a significant issue in home-based rehabilitation. Automated movement quality assessment (MQA) using skeletal movement data (hereafter referred to as skeletal data) collected via depth imaging devices can assist with home-based rehabilitation by providing the necessary quantitative feedback. This paper aims to use recent advances in deep learning to address the problem of MQA. Movement quality score generation is an essentia…
Attack Assessment and Augmented Identity Recognition for Human Skeleton Data
Dept. of ECE（电子工程系）；Old Dominion University（旧 Dominion 大学）；Dept. of Computer Science（计算机科学系）；Amherst College（阿默斯特学院）；Hampshire County, MA, USA（马萨诸塞州汉普郡县）
The Pennsylvania State University（宾夕法尼亚州立大学）；Johns Hopkins University（约翰斯·霍普金斯大学）；Shenzhen Baoan Women’s and Children’s Hospital, Jinan University（暨南大学深圳宝安妇幼保健院）；Xi’an University of Architecture and Technology（西安建筑科技大学）
General movement assessment (GMA) of infant movement videos (IMVs) is an effective method for early detection of cerebral palsy (CP) in infants. We demonstrate in this paper that end-to-end trainable neural networks for image sequence recognition can be applied to achieve good results in GMA, and more importantly, augmenting raw video with infant body parsing and pose estimation information can significantly improve performance. To solve the problem of efficiently utilizing partially labeled IMVs for body parsing, we propose a semi-supervised model, termed SiamParseNet (SPN), which consists of…
Omnidirectional images and videos can provide immersive experience of real-world scenes in Virtual Reality (VR) environment. We present a perceptual omnidirectional image quality assessment (IQA) study in this paper since it is extremely important to provide a good quality of experience under the VR environment. We first establish an omnidirectional IQA (OIQA) database, which includes 16 source images and 320 distorted images degraded by 4 commonly encountered distortion types, namely JPEG compression, JPEG2000 compression, Gaussian blur and Gaussian noise. Then a subjective quality evaluation…
Cerebral Palsy Alliance Research Institute（脑瘫联盟研究所）；The University of Sydney（悉尼大学）；The Children’s Hospital at Westmead（韦斯特米德儿童医院）
The absence or abnormality of fidgety movements of joints or limbs is strongly indicative of cerebral palsy in infants. Developing computer-based methods for assessing infant movements in videos is pivotal for improved cerebral palsy screening. Most existing methods use appearance-based features and are thus sensitive to strong but irrelevant signals caused by background clutter or a moving camera. Moreover, these features are computed over the whole frame, thus they measure gross whole body movements rather than specific joint/limb motion. Addressing these challenges, we develop and validate …
Two-Stage Multi-Modal Fusion with Adaptive Alignment for Action Quality Assessment
Tsinghua University（清华大学）；Beihang University（北京航空航天大学）；Children’s Hospital, Capital Institute of Pediatrics（首都儿科研究所附属儿童医院）；Zhongguancun Laboratory（中关村实验室）
Towards Scalable Newborn Screening: Automated General Movement Assessment in Uncontrolled Settings
ETH Zurich（苏黎世联邦理工学院）；University of Zurich（苏黎世大学）；University of Regensburg（莱茵河畔埃尔兰根大学）；University Children’s Hospital（儿童医院）；University Children’s Hospital Regensburg（埃尔兰根儿童医院）
Recent advances in tracking sensors and pose estimation software enable smart systems to use trajectories of skeleton joint locations for supervised learning. We study the problem of accurately recognizing sign language words, which is key to narrowing the communication gap between hard and non-hard of hearing people. Our method explores a geometric feature space that we call `sub-skeleton' aspects of movement. We assess similarity of feature space trajectories using natural, speed invariant distance measures, which enables clear and insightful nearest neighbor classification. The simplicity a…
Open, or non-laparoscopic surgery, represents the vast majority of all operating room procedures, but few tools exist to objectively evaluate these techniques at scale. Current efforts involve human expert-based visual assessment. We leverage advances in computer vision to introduce an automated approach to video analysis of surgical execution. A state-of-the-art convolutional neural network architecture for object detection was used to detect operating hands in open surgery videos. Automated assessment was expanded by combining model predictions with a fast object tracker to enable surgeon-sp…
Parkinson's Disease Assessment from a Wrist-Worn Wearable Sensor in Free-Living Conditions: Deep Ensemble Learning and Visualization
University of Waterloo（滑铁卢大学）；Ludwig-Maximilians University（路德维希-马克西米利安大学）；Technical University of Munich（慕尼黑工业大学）；Schön Klinik München Schwabing（慕尼黑施瓦宾 Schön 诊所）
Toward Personalized Darts Training: A Data-Driven Framework Based on Skeleton-Based Biomechanical Analysis and Motion Modeling
School of Computer Technology and Application, Qinghai University（青海大学计算机技术与应用学院）；Xiaoping Technology Innovation Lab, Zhongshan Xiaolan Senior High School（中山市小榄中学小平科技创新实验室）；School of Artificial Intelligence, Chongqing University of Technology（重庆理工大学人工智能学院）；Department of Language Science and Technology, The Hong Kong Polytechnic University（香港理工大学语言科学与技术系）；Faculty of Education, University of Macau（澳门大学教育学院）
Institute of Electronics and Computer Science (EDI)（电子与计算机科学研究所（EDI））；Riga Stradins University（里加斯特拉迪什大学）
Washing hands is one of the most important ways to prevent infectious diseases, including COVID-19. Unfortunately, medical staff does not always follow the World Health Organization (WHO) hand washing guidelines in their everyday work. To this end, we present neural networks for automatically recognizing the different washing movements defined by the WHO. We train the neural network on a part of a large (2000+ videos) real-world labeled dataset with the different washing movements. The preliminary results show that using pre-trained neural network models such as MobileNetV2 and Xception for th…
TU Delft（代尔夫特理工大学）；Instituto Tecnologico de Costa Rica（哥斯达黎加技术学院）
Laughter is considered one of the most overt signals of joy. Laughter is well-recognized as a multimodal phenomenon but is most commonly detected by sensing the sound of laughter. It is unclear how perception and annotation of laughter differ when annotated from other modalities like video, via the body movements of laughter. In this paper we take a first step in this direction by asking if and how well laughter can be annotated when only audio, only video (containing full body movement information) or audiovisual modalities are available to annotators. We ask whether annotations of laughter a…
