This paper proposes a reliable approach for human gait symmetry assessment using a depth camera and two mirrors. The input of our system is a sequence of 3D point clouds which are formed from a setup including a Time-of-Flight (ToF) depth camera and two mirrors. A cylindrical histogram is estimated for describing the posture in each point cloud. The sequence of such histograms is then separated into two sequences of sub-histograms representing two half-bodies. A cross-correlation technique is finally applied to provide values describing gait symmetry indices. The evaluation was performed on 9 different gait types to demonstrate the ability of our approach in assessing gait symmetry. A comparison between our system and related methods, that employ different input data types, is also provided.
The University of Hong Kong（香港大学）；Southern University of Science and Technology（南方科技大学）；The Hong Kong Polytechnic University（香港理工大学）
Video-based gait recognition has achieved impressive results in constrained scenarios. However, visual cameras neglect human 3D structure information, which limits the feasibility of gait recognition in the 3D wild world. Instead of extracting gait features from images, this work explores precise 3D gait features from point clouds and proposes a simple yet efficient 3D gait recognition framework, termed LidarGait. Our proposed approach projects sparse point clouds into depth maps to learn the representations with 3D geometry information, which outperforms existing point-wise and camera-based m…
University of Montreal（蒙特利尔大学）；University of Science and Technology（科技大学）
Reconstructing 3D object models is playing an important role in many applications in the field of computer vision. Instead of employing a collection of cameras and/or sensors as in many studies, this paper proposes a simple way to build a cheaper system for 3D reconstruction using only one depth camera and 2 or more mirrors. Each mirror is equivalently considered as a depth camera at another viewpoint. Since all scene data are provided by only one depth sensor, our approach can be applied to moving objects and does not require any synchronization protocol as with a set of cameras. Some experim…
Text-guided Feature Disentanglement for Cross-modal Gait Recognition
Fujian Key Laboratory of Urban Intelligent Sensing and Computing, Xiamen University（福建城市智能感知与计算重点实验室，厦门大学）；Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University（多媒体可信感知与高效计算重点实验室，中华人民共和国教育部，厦门大学）
Zhejiang University（浙江大学）；Key Laboratory of Collaborative Sensing and Autonomous Unmanned Systems of Zhejiang Province（浙江省协同感知与自主无人系统重点实验室）
Gait recognition is widely used in diversified practical applications. Currently, the most prevalent approach is to recognize human gait from RGB images, owing to the progress of computer vision technologies. Nevertheless, the perception capability of RGB cameras deteriorates in rough circumstances, and visual surveillance may cause privacy invasion. Due to the robustness and non-invasive feature of millimeter wave (mmWave) radar, radar-based gait recognition has attracted increasing attention in recent years. In this research, we propose a Hierarchical Dynamic Network (HDNet) for gait recogni…
University of Montreal（蒙特利尔大学）；University of Science and Technology（科技大学）
This paper presents an initial work on assessment of gait normality in which the human body motion is represented by a sequence of enhanced depth maps. The input data is provided by a system consisting of a Time-of-Flight (ToF) depth camera and two mirrors. This approach proposes two feature types to describe characteristics of localized points of interest and the level of posture symmetry. These two features are processed on a sequence of enhanced depth maps with the support of a sliding window to provide two corresponding scores. The gait assessment is finally performed based on a weighted c…
A Multi-View High-Resolution Foot-Ankle Complex Point Cloud Dataset During Gait for Occlusion-Robust 3D Completion
College of Biomedical Engineering（生物医学工程学院）；College of Computer Science and Artificial Intelligence（计算机科学与人工智能学院）；Department of Information Engineering（信息工程系）；School of Computer Science and Engineering（计算机科学与工程学院）
ShanghaiTech University（上海科技大学）；Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences（中国科学院上海微系统与信息技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Shanghai Engineering Research Center of Intelligent Vision and Imaging（上海智能视觉与成像工程技术研究中心）
Photo-realistic neural reconstruction and rendering of the human portrait are critical for numerous VR/AR applications. Still, existing solutions inherently rely on multi-view capture settings, and the one-shot solution to get rid of the tedious multi-view synchronization and calibration remains extremely challenging. In this paper, we propose MirrorNeRF - a one-shot neural portrait free-viewpoint rendering approach using a catadioptric imaging system with multiple sphere mirrors and a single high-resolution digital camera, which is the first to combine neural radiance field with catadioptric …
Universidad Politécnica de Madrid（马德里理工大学）；Centre for Automation and Robotics (CAR UPM-CSIC)（自动化与机器人中心（马德里理工大学-西班牙国家研究委员会））；ETSIDI（马德里理工大学工业设计工程学院）
The use of 3D cameras for gait analysis has been highly questioned due to the low accuracy they have demonstrated in the past. The objective of the study presented in this paper is to improve the accuracy of the estimations made by robot-mounted 3D cameras in human gait analysis by applying a supervised learning stage. The 3D camera was mounted in a mobile robot to obtain a longer walking distance. This study shows an improvement in detection of kinematic gait signals and gait descriptors by post-processing the raw estimations of the camera using artificial neural networks trained with the dat…
The use of gait for person identification has important advantages such as being non-invasive, unobtrusive, not requiring cooperation and being less likely to be obscured compared to other biometrics. Existing methods for gait recognition require cooperative gait scenarios, in which a single person is walking multiple times in a straight line in front of a camera. We aim to address the challenges of real-world scenarios in which camera feeds capture multiple people, who in most cases pass in front of the camera only once. We address privacy concerns by using only motion information of walking …
Institute of Image Processing and Network Engineering, Shanghai Jiao Tong University（上海交通大学图像处理与网络工程研究所）；School of Biomedical Engineering, Shanghai Jiao Tong University（上海交通大学生物医学工程学院）；Shanghai Key Laboratory of Scalable Computing and Systems, Shanghai Jiao Tong University（上海交通大学可扩展计算与系统重点实验室）
ideo-based person re-identification (Re-ID) aims to match person images in video sequences captured by disjoint surveillance cameras. Traditional video-based person Re-ID methods focus on exploring appearance information, thus, vulnerable against illumination changes, scene noises, camera parameters, and especially clothes/carrying variations. Gait recognition provides an implicit biometric solution to alleviate the above headache. Nonetheless, it experiences severe performance degeneration as camera view varies. In an attempt to address these problems, in this paper, we propose a framework th…
Obtaining demographics information from video is valuable for a range of real-world applications. While approaches that leverage facial features for gender inference are very successful in restrained environments, they do not work in most real-world scenarios when the subject is not facing the camera, has the face obstructed or the face is not clear due to distance from the camera or poor resolution. We propose a weakly-supervised method for learning gender information of people based on their manner of walking. We make use of state-of-the art facial analysis models to automatically annotate f…
Southeast University（东南大学）；University of Minnesota（明尼苏达大学）；Singapore University of Technology and Design（新加坡科技设计大学）；The University of Edinburgh（爱丁堡大学）
Human identification is a key requirement for many applications in everyday life, such as personalized services, automatic surveillance, continuous authentication, and contact tracing during pandemics, etc. This work studies the problem of cross-modal human re-identification (ReID), in response to the regular human movements across camera-allowed regions (e.g., streets) and camera-restricted regions (e.g., offices) deployed with heterogeneous sensors. By leveraging the emerging low-cost RGB-D cameras and mmWave radars, we propose the first-of-its-kind vision-RF system for cross-modal multi-per…
Cross-Modality Gait Recognition: Bridging LiDAR and Camera Modalities for Human Identification
Southern University of Science and Technology（南方科技大学）；The University of Hong Kong（香港大学）；University of Cordoba（科尔多瓦大学）；The Hong Kong Polytechnic University（香港理工大学）
University of Oxford（牛津大学）；University of Glasgow（格拉斯哥大学）；Shanghai Jiao Tong University（上海交通大学）
With the increasing awareness of high-quality life, there is a growing need for health monitoring devices running robust algorithms in home environment. Health monitoring technologies enable real-time analysis of users' health status, offering long-term healthcare support and reducing hospitalization time. The purpose of this work is twofold, the software focuses on the analysis of gait, which is widely adopted for joint correction and assessing any lower limb or spinal problem. On the hardware side, we design a novel marker-less gait analysis device using a low-cost RGB camera mounted on a mo…
Nanyang Technological University（南洋理工大学）；University of California, Berkeley（加州大学伯克利分校）；University of Edinburgh（爱丁堡大学）
As an important biomarker for human identification, human gait can be collected at a distance by passive sensors without subject cooperation, which plays an essential role in crime prevention, security detection and other human identification applications. At present, most research works are based on cameras and computer vision techniques to perform gait recognition. However, vision-based methods are not reliable when confronting poor illuminations, leading to degrading performances. In this paper, we propose a novel multimodal gait recognition method, namely GaitFi, which leverages WiFi signa…
McGill University（麦吉尔大学）；McGill University Health Centre（麦吉尔大学健康中心）；Montreal Neurological Institute and Hospital（蒙特利尔神经学研究所及医院）；AreteX Engineering（AreteX工程公司）
Gait analysis of patients with neurological disorders, including multiple sclerosis (MS), is important for rehabilitation and treatment. The Mircrosoft Kinect sensor, which was developed for motion recognition in gaming applications, is an ideal candidate for an inexpensive system providing the capability for human gait analysis. In this research, we develop a framework to quantify the gait abnormality of MS patients using a Kinect for Windows camera. In addition to the previously introduced gait indices, a novel set of MS gait indices based on the concept of dynamic time warping is introduced…
SRM Institute of Science and Technology（SRM科学技术学院）；Instituto Superior Técnico, Univ. of Lisbon（里斯本大学高等技术学院）
Biometrics plays a significant role in vision-based surveillance applications. Soft biometrics such as gait is widely used with face in surveillance tasks like person recognition and re-identification. Nevertheless, in practical scenarios, classical fusion techniques respond poorly to changes in individual users and in the external environment. To this end, we propose a novel adaptive multi-biometric fusion strategy for the dynamic incorporation of gait and face biometric cues by leveraging keyless attention deep neural networks. Various external factors such as viewpoint and distance to the c…
University of Texas at Arlington（德克萨斯大学阿灵顿分校）；Yale University（耶鲁大学）
As mobile technologies have become ubiquitous in recent years, computer-based cognitive tests have become more popular and efficient. In this work, we focus on assessing motor function in children by analyzing their gait movements. Although there has been a lot of research on designing automated assessment systems for gait analysis, most of these efforts use obtrusive wearable sensors for measuring body movements. We have devised a computer vision-based assessment system that only requires a camera which makes it easier to employ in school or home environments. A dataset has been created with …
Hangzhou Dianzi University（杭州电子科技大学）；Explore Academy of JD.com（京东探索研究院）
Existing studies for gait recognition are dominated by 2D representations like the silhouette or skeleton of the human body in constrained scenes. However, humans live and walk in the unconstrained 3D space, so projecting the 3D human body onto the 2D plane will discard a lot of crucial information like the viewpoint, shape, and dynamics for gait recognition. Therefore, this paper aims to explore dense 3D representations for gait recognition in the wild, which is a practical yet neglected problem. In particular, we propose a novel framework to explore the 3D Skinned Multi-Person Linear (SMPL) …
University of Bristol（布里斯托大学）；The Australian National University（澳大利亚国立大学）；The University of Hong Kong（香港大学）
This paper addresses the problem of mirror surface reconstruction, and proposes a solution based on observing the reflections of a moving reference plane on the mirror surface. Unlike previous approaches which require tedious calibration, our method can recover the camera intrinsics, the poses of the reference plane, as well as the mirror surface from the observed reflections of the reference plane under at least three unknown distinct poses. We first show that the 3D poses of the reference plane can be estimated from the reflection correspondences established between the images and the refere…
Daffodil International University（达法迪尔国际大学）；University of Calgary（卡尔加里大学）
Walking is one of the most common modes of terrestrial locomotion for humans. Walking is essential for humans to perform most kinds of daily activities. When a person walks, there is a pattern in it, and it is known as gait. Gait analysis is used in sports and healthcare. We can analyze this gait in different ways, like using video captured by the surveillance cameras or depth image cameras in the lab environment. It also can be recognized by wearable sensors. e.g., accelerometer, force sensors, gyroscope, flexible goniometer, magneto resistive sensors, electromagnetic tracking system, force s…
Human gait is one of important biometric characteristics for human identification at a distance. In practice, occlusion usually occurs and seriously affects accuracy of gait recognition. However, there is no available database to support in-depth research of this problem, and state-of-arts gait recognition methods have not paid enough attention to it, thus this paper focuses on gait recognition under occlusion. We collect a new gait recognition database called OG RGB+D database, which breaks through the limitation of other gait databases and includes multimodal gait data of various occlusions …
Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Tencent AI LAB（腾讯人工智能实验室）；Northeast Forestry University（东北林业大学）
The model-based gait recognition methods usually adopt the pedestrian walking postures to identify human beings. However, existing methods did not explicitly resolve the large intra-class variance of human pose due to camera views changing. In this paper, we propose to generate multi-view pose sequences for each single-view pose sample by learning full-rank transformation matrices via lower-upper generative adversarial network (LUGAN). By the prior of camera imaging, we derive that the spatial coordinates between cross-view poses satisfy a linear transformation of a full-rank matrix, thereby, …
This paper proposes a novel algorithm of discovering the structure of a kaleidoscopic imaging system that consists of multiple planar mirrors and a camera. The kaleidoscopic imaging system can be recognized as the virtual multi-camera system and has strong advantages in that the virtual cameras are strictly synchronized and have the same intrinsic parameters. In this paper, we focus on the extrinsic calibration of the virtual multi-camera system. The problems to be solved in this paper are two-fold. The first problem is to identify to which mirror chamber each of the 2D projections of mirrored…
Queensland University of Technology（昆士兰科技大学）；Australian Centre for Robotic Vision（澳大利亚机器人视觉中心）；Stanford University（斯坦福大学）
This paper proposes the design of a custom mirror-based light field camera adapter that is cheap, simple in construction, and accessible. Mirrors of different shape and orientation reflect the scene into an upwards-facing camera to create an array of virtual cameras with overlapping field of view at specified depths, and deliver video frame rate light fields. We describe the design, construction, decoding and calibration processes of our mirror-based light field camera adapter in preparation for an open-source release to benefit the robotic vision community.
Improved mirror ball projection for more accurate merging of multiple camera outputs and process monitoring
DMG MORI CO. LTD.（德马吉森精机有限公司）；Fraunhofer-Institut für Werkstoff- und Strahltechnik（弗劳恩霍夫材料与光束技术研究所）；Berufsakademie Sachsen（萨克森职业学院）；Staatliche Studienakademie Dresden（德累斯顿国立应用技术大学）
In applications such as autonomous driving, it is important to understand, infer, and anticipate the intention and future behavior of pedestrians. This ability allows vehicles to avoid collisions and improve ride safety and quality. This paper proposes a biomechanically inspired recurrent neural network (Bio-LSTM) that can predict the location and 3D articulated body pose of pedestrians in a global coordinate frame, given 3D poses and locations estimated in prior frames with inaccuracy. The proposed network is able to predict poses and global locations for multiple pedestrians simultaneously, …
University of Zagreb Faculty of Electrical Engineering and Computing（萨格勒布大学电气工程与计算学院）
We present a 3D printed adapter with planar mirrors for stereo reconstruction using front and back smartphone camera. The adapter presents a practical and low-cost solution for enabling any smartphone to be used as a stereo camera, which is currently only possible using high-end phones with expensive 3D sensors. Using the prototype version of the adapter, we experiment with parameters like the angles between cameras and mirrors and the distance to each camera (the stereo baseline). We find the most convenient configuration and calibrate the stereo pair. Based on the presented preliminary analy…
University of Bristol（布里斯托大学）；Southmead Hospital（南米德医院）
Evaluating neurological disorders such as Parkinson's disease (PD) is a challenging task that requires the assessment of several motor and non-motor functions. In this paper, we present an end-to-end deep learning framework to measure PD severity in two important components, hand movement and gait, of the Unified Parkinson's Disease Rating Scale (UPDRS). Our method leverages on an Inflated 3D CNN trained by a temporal segment framework to learn spatial and long temporal structure in video data. We also deploy a temporal attention mechanism to boost the performance of our model. Further, motion…
Institute for Systems and Robotics, Instituto Superior Técnico, Universidade de Lisboa（里斯本大学高等技术学院系统与机器人研究所）
In this article we purpose a novel method for planar pose estimation of mobile robots. This method is based on an analytic solution (which we derived) for the projection of 3D straight lines, onto the mirror of Non-Central Catadioptric Cameras (NCCS). The resulting solution is rewritten as a function of the rotation and translation parameters, which is then used as an error function for a set of mirror points. Those should be the result of the projection of a set of points incident with the respective 3D lines. The camera's pose is given by minimizing the error function, with the associated co…
Escola Bahiana de Medicina e Saúde Pública（巴伊亚纳医学与公共卫生学院）；Dass Nordeste Calçados e Artigos Esportivos S.A（Dass Nordeste鞋类及体育用品股份有限公司）
Despite the evaluation systems of human movement that have been advancing in recent decades, their use are not feasible for clinical practice because it has a high cost and scarcity of trained operators to interpret their results. An ideal videogrammetry system should be easy to use, low cost, with minimal equipment, and fast realization. The CvMob is a free tool for dynamic evaluation of human movements that express measurements in figures, tables, and graphics. This paper aims to determine if CvMob is a reliable tool for the evaluation of two dimensional human gait. This is a validity and re…
CEA, IRFU, Service d’Astrophysique-SEDI（法国原子能和替代能源委员会、核研究与粒子物理研究所、天体物理服务部）
We propose an original concept of compressive sensing (CS) polarimetric imaging based on a digital micro-mirror (DMD) array and two single-pixel detectors. The polarimetric sensitivity of the proposed setup is due to an experimental imperfection of reflecting mirrors which is exploited here to form an original reconstruction problem, including a CS problem and a source separation task. We show that a two-step approach tackling each problem successively is outperformed by a dedicated combined reconstruction method, which is explicited in this article and preferably implemented through a reweigh…
