This paper proposes an approach estimating a gait abnormality index based on skeletal information provided by a depth camera. Differently from related works where the extraction of hand-crafted features is required to describe gait characteristics, our method automatically performs that stage with the support of a deep auto-encoder. In order to get visually interpretable features, we embedded a constraint of sparsity into the model. Similarly to most gait-related studies, the temporal factor is also considered as a post-processing in our system. This method provided promising results when experimenting on a dataset containing nearly one hundred thousand skeleton samples.
In this paper, we propose a method that estimates a gait index for a sequence of skeletons. Our system is a stack of an encoder and a decoder that are formed by Long Short-Term Memories (LSTMs). In the encoding stage, the characteristics of an input are automatically determined and are compressed into a latent space. The decoding stage then attempts to reconstruct the input according to such intermediate representation. The reconstruction error is thus considered as a weak gait index. By combining such weak indices over a long-time movement, our system can provide a good estimation for the gai…
DAGait: Generalized Skeleton-Guided Data Alignment for Gait Recognition
The Shenzhen International Graduate School, Tsinghua University Shenzhen, China（深圳国际研究生院，清华大学深圳学院中国）；School of Computer Science（计算机科学学院）；Information Engineering, Hefei University of Technology Hefei, China（信息工程，合肥工业大学合肥中国）
The use of gait for person identification has important advantages such as being non-invasive, unobtrusive, not requiring cooperation and being less likely to be obscured compared to other biometrics. Existing methods for gait recognition require cooperative gait scenarios, in which a single person is walking multiple times in a straight line in front of a camera. We aim to address the challenges of real-world scenarios in which camera feeds capture multiple people, who in most cases pass in front of the camera only once. We address privacy concerns by using only motion information of walking …
Human gait is one of important biometric characteristics for human identification at a distance. In practice, occlusion usually occurs and seriously affects accuracy of gait recognition. However, there is no available database to support in-depth research of this problem, and state-of-arts gait recognition methods have not paid enough attention to it, thus this paper focuses on gait recognition under occlusion. We collect a new gait recognition database called OG RGB+D database, which breaks through the limitation of other gait databases and includes multimodal gait data of various occlusions …
Benchmarking Skeleton-based Motion Encoder Models for Clinical Applications: Estimating Parkinson's Disease Severity in Walking Sequences
University of Toronto（多伦多大学）；Toronto Rehabilitation Institute – University Health Network（多伦多康复研究所 - 大学健康网络）；Vienna University of Technology（维也纳工业大学）；Center for Mental Health – University Health Network（心理健康中心 - 大学健康网络）
OpenGait: A Comprehensive Benchmark Study for Gait Recognition towards Better Practicality
School of Artificial Intelligence, Shenzhen University (SZU)（深圳大学人工智能学院）；National Engineering Laboratory for Big Data System Computing Technology, SZU（大数据系统计算技术国家工程实验室）；Southern University of Science and Technology (SUSTech)（南方科技大学）；School of Artificial Intelligence, Beijing Normal University（北京师范大学人工智能学院）；Watrix Technology Limited Co. Ltd（纬思科技有限公司）；Department of Computer Vision Technology (VIS), Baidu Inc.（百度公司计算机视觉技术部）；Shenzhen Institute of Advanced Study, University of Electronic Science and Technology of China (UESTC)（中国电子科学研究院深圳研究院）；Institute of Applied Artificial Intelligence of the Guangdong-Hong Kong-Macao Greater Bay Area, Shenzhen Polytechnic University（粤港澳大湾区应用人工智能研究院，深圳职业技术大学）；Department of Computer Science and Engineering, Southern University of Science and Technology (SUSTech)（南方科技大学计算机科学与工程系）
This paper proposes an approach that estimates human walking gait quality index using an adversarial auto-encoder (AAE), i.e. a combination of auto-encoder and generative adversarial network (GAN). Since most GAN-based models have been employed as data generators, our work introduces another perspective of their application. This method directly works on a sequence of 3D point clouds representing the walking postures of a subject. By fitting a cylinder onto each point cloud and feeding obtained histograms to an appropriate AAE, our system is able to provide different measures that may be used …
Hangzhou Dianzi University（杭州电子科技大学）；Explore Academy of JD.com（京东探索研究院）
Existing studies for gait recognition are dominated by 2D representations like the silhouette or skeleton of the human body in constrained scenes. However, humans live and walk in the unconstrained 3D space, so projecting the 3D human body onto the 2D plane will discard a lot of crucial information like the viewpoint, shape, and dynamics for gait recognition. Therefore, this paper aims to explore dense 3D representations for gait recognition in the wild, which is a practical yet neglected problem. In particular, we propose a novel framework to explore the 3D Skinned Multi-Person Linear (SMPL) …
Imperial College London（伦敦帝国理工学院）；Shanghai Jiao Tong University（上海交通大学）
Accurate estimation of three-dimensional human skeletons from depth images can provide important metrics for healthcare applications, especially for biomechanical gait analysis. However, there exist inherent problems associated with depth images captured from a single view. The collected data is greatly affected by occlusions where only partial surface data can be recorded. Furthermore, depth images of human body exhibit heterogeneous characteristics with viewpoint changes, and the estimated poses under local coordinate systems are expected to go through equivariant rotations. Most existing po…
University of North Carolina at Charlotte（北卡罗来纳大学夏洛特分校）；University of Central Florida（中佛罗里达大学）
Most existing gait recognition methods are appearance-based, which rely on the silhouettes extracted from the video data of human walking activities. The less-investigated skeleton-based gait recognition methods directly learn the gait dynamics from 2D/3D human skeleton sequences, which are theoretically more robust solutions in the presence of appearance changes caused by clothes, hairstyles, and carrying objects. However, the performance of skeleton-based solutions is still largely behind the appearance-based ones. This paper aims to close such performance gap by proposing a novel network mo…
Skeleton-based gait recognition models usually suffer from the robustness problem, as the Rank-1 accuracy varies from 90\% in normal walking cases to 70\% in walking with coats cases. In this work, we propose a state-of-the-art robust skeleton-based gait recognition model called Gait-TR, which is based on the combination of spatial transformer frameworks and temporal convolutional networks. Gait-TR achieves substantial improvements over other skeleton-based gait models with higher accuracy and better robustness on the well-known gait dataset CASIA-B. Particularly in walking with coats cases, G…
Gait recognition captures gait patterns from the walking sequence of an individual for identification. Most existing gait recognition methods learn features from silhouettes or skeletons for the robustness to clothing, carrying, and other exterior factors. The combination of the two data modalities, however, is not fully exploited. Previous multimodal gait recognition methods mainly employ the skeleton to assist the local feature extraction where the intrinsic discrimination of the skeleton data is ignored. This paper proposes a simple yet effective Bimodal Fusion (BiFusion) network which mine…
College of Computer Science and Technology, Qingdao University（青岛大学计算机科学技术学院）；College of Information Science and Engineering, Ocean University of China（中国海洋大学信息科学与工程学院）；School of Informatics, University of Leicester（莱斯特大学信息学院）
It is a challenging task to identify a person based on her/his gait patterns. State-of-the-art approaches rely on the analysis of temporal or spatial characteristics of gait, and gait recognition is usually performed on single modality data (such as images, skeleton joint coordinates, or force signals). Evidence has shown that using multi-modality data is more conducive to gait research. Therefore, we here establish an automated learning system, with an associated spatio-temporal capsule network (ASTCapsNet) trained on multi-sensor datasets, to analyze multimodal information for gait recogniti…
Gait recognition is a promising video-based biometric for identifying individual walking patterns from a long distance. At present, most gait recognition methods use silhouette images to represent a person in each frame. However, silhouette images can lose fine-grained spatial information, and most papers do not regard how to obtain these silhouettes in complex scenes. Furthermore, silhouette images contain not only gait features but also other visual clues that can be recognized. Hence these approaches can not be considered as strict gait recognition. We leverage recent advances in human pose…
Gait recognition is a promising biometric with unique properties for identifying individuals from a long distance by their walking patterns. In recent years, most gait recognition methods used the person's silhouette to extract the gait features. However, silhouette images can lose fine-grained spatial information, suffer from (self) occlusion, and be challenging to obtain in real-world scenarios. Furthermore, these silhouettes also contain other visual clues that are not actual gait features and can be used for identification, but also to fool the system. Model-based methods do not suffer fro…
The Shenzhen International Graduate School, Tsinghua University, China（清华大学深圳国际研究生院）；School of Computer Science and Information Engineering, Hefei University of Technology, China（合肥工业大学计算机科学与信息工程学院）
Gait recognition, a long-distance biometric technology, has aroused intense interest recently. Currently, the two dominant gait recognition works are appearance-based and model-based, which extract features from silhouettes and skeletons, respectively. However, appearance-based methods are greatly affected by clothes-changing and carrying conditions, while model-based methods are limited by the accuracy of pose estimation. To tackle this challenge, a simple yet effective two-branch network is proposed in this paper, which contains a CNN-based branch taking silhouettes as input and a GCN-based …
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；South China University of Technology（华南理工大学）；The Chinese University of Hong Kong（香港中文大学）；National University of Defense Technology（国防科技大学）；Lanzhou University（兰州大学）
Gait-based person re-identification (Re-ID) is valuable for safety-critical applications, and using only 3D skeleton data to extract discriminative gait features for person Re-ID is an emerging open topic. Existing methods either adopt hand-crafted features or learn gait features by traditional supervised learning paradigms. Unlike previous methods, we for the first time propose a generic gait encoding approach that can utilize unlabeled skeleton data to learn gait representations in a self-supervised manner. Specifically, we first propose to introduce self-supervision by learning to reconstru…
School of information Science and Engineering, Lanzhou University（兰州大学信息科学与工程学院）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Institute of Engineering Medicine, Beijing Institute of Technology（北京理工大学医学工程学院）
In recent years, the incidence of depression is rising rapidly worldwide, but large-scale depression screening is still challenging. Gait analysis provides a non-contact, low-cost, and efficient early screening method for depression. However, the early screening of depression based on gait analysis lacks sufficient effective sample data. In this paper, we propose a skeleton data augmentation method for assessing the risk of depression. First, we propose five techniques to augment skeleton data and apply them to depression and emotion datasets. Then, we divide augmentation methods into two type…
Recognizing the human emotion automatically from visual characteristics plays a vital role in many intelligent applications. Recently, gait-based emotion recognition, especially gait skeletons-based characteristic, has attracted much attention, while many available methods have been proposed gradually. The popular pipeline is to first extract affective features from joint skeletons, and then aggregate the skeleton joint and affective features as the feature vector for classifying the emotion. However, the aggregation procedure of these emerged methods might be rigid, resulting in insufficientl…
School of Cyber Science and Engineering, Shanghai Jiao Tong University（上海交通大学网络空间安全学院）
As an emerging biological identification technology, vision-based gait identification is an important research content in biometrics. Most existing gait identification methods extract features from gait videos and identify a probe sample by a query in the gallery. However, video data contains redundant information and can be easily influenced by bagging (BG) and clothing (CL). Since human body skeletons convey essential information about human gaits, a skeleton-based gait identification network is proposed in our project. First, extract skeleton sequences from the video and map them into a gai…
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；National University of Defense Technology（国防科技大学）；Lanzhou University（兰州大学）；South China University of Technology（华南理工大学）；Second Clinical Medical College, Jinan University（暨南大学第二临床医学院）；Beijing Institute of Technology（北京理工大学）
Person re-identification (Re-ID) via gait features within 3D skeleton sequences is a newly-emerging topic with several advantages. Existing solutions either rely on hand-crafted descriptors or supervised gait representation learning. This paper proposes a self-supervised gait encoding approach that can leverage unlabeled skeleton data to learn gait representations for person Re-ID. Specifically, we first create self-supervision by learning to reconstruct unlabeled skeleton sequences reversely, which involves richer high-level semantics to obtain better gait representations. Other pretext tasks…
University of Virginia（弗吉尼亚大学）；Old Dominion University（奥多明尼昂大学）
Gait recognition using noninvasively acquired data has been attracting an increasing interest in the last decade. Among various modalities of data sources, it is experimentally found that the data involving skeletal representation are amenable for reliable feature compaction and fast processing. Model-based gait recognition methods that exploit features from a fitted model, like skeleton, are recognized for their view and scale-invariant properties. We propose a model-based gait recognition method, using sequences recorded by a single flash lidar. Existing state-of-the-art model-based approach…
Single-Stream Multi-Feature Fusion with Temporal Robustness for Gait Emotion Recognition
College of Computer and Information Science, Southwest University（西南大学计算机与信息科学学院）；College of Physical Science and Technology, Southwest University（西南大学物理科学与技术学院）；Wisesoft Inc.（四川川大智胜软件股份有限公司）
Motif Guided Graph Transformer with Combinatorial Skeleton Prototype Learning for Skeleton-Based Person Re-Identification
College of Computing and Data Science, Nanyang Technological University (NTU)（南洋理工大学计算与数据科学学院）；Joint NTU-UBC Research Centre of Excellence in Active Living for the Elderly (LILY)（南洋理工大学-不列颠哥伦比亚大学老年活跃生活卓越联合研究中心）
A Multi-Stage Adaptive Feature Fusion Neural Network for Multimodal Gait Recognition
Central South University（中南大学）；Southern University of Science and Technology（南方科技大学）；Research Institute of Trustworthy Autonomous System, Southern University of Science and Technology（南方科技大学可信自主系统研究院）
SRM Institute of Science and Technology（SRM科学技术研究所）；CMR Institute of Technology（CMR理工学院）
Human identification plays a prominent role in terms of security. In modern times security is becoming the key term for an individual or a country, especially for countries which are facing internal or external threats. Gait analysis is interpreted as the systematic study of the locomotive in humans. It can be used to extract the exact walking features of individuals. Walking features depends on biological as well as the physical feature of the object; hence, it is unique to every individual. In this work, gait features are used to identify an individual. The steps involve object detection, ba…
Affinity Contrastive Learning for Skeleton-based Human Activity Understanding
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；School of Electronic, Electrical and Communication Engineering, Univeristy of Chinese Academy of Sciences（中国科学院大学电子电气与通信工程学院）；School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）；Moonshot AI
An Elastic Shape Variational Autoencoder for Skeleton Pose Trajectories
Systems and Information Engineering, University of Virginia（弗吉尼亚大学系统与信息工程系）；Biomedical Engineering, Johns Hopkins University（约翰霍普金斯大学生物医学工程系）；Dept. of Applied Mathematics and Statistics, Johns Hopkins University（约翰霍普金斯大学应用数学与统计学系）
Overcoming the visual barrier and developing "see-through vision" has been one of mankind's long-standing dreams. Unlike visible light, Radio Frequency (RF) signals penetrate opaque obstructions and reflect highly off humans. This paper establishes a deep-learning model that can be trained to reconstruct continuous video of a 15-point human skeleton even through visual occlusion. The training process adopts a student/teacher learning procedure inspired by the Feynman learning technique, in which video frames and RF data are first collected simultaneously using a co-located setup containing an …
