The Human-Machine Interaction (HMI) research field is an important topic in machine learning that has been deeply investigated thanks to the rise of computing power in the last years. The first time, it is possible to use machine learning to classify images and/or videos instead of the traditional computer vision algorithms. The aim of this paper is to build a symbiosis between a convolutional neural network (CNN) and a recurrent neural network (RNN) to recognize cultural/anthropological Italian sign language gestures from videos. The CNN extracts important features that later are used by the RNN. With RNNs we are able to store temporal information inside the model to provide contextual information from previous frames to enhance the prediction accuracy. Our novel approach uses different data augmentation techniques and regularization methods from only RGB frames to avoid overfitting and provide a small generalization error.
Due to the universal non-verbal natural communication approach that allows for effective communication between humans, gesture recognition technology has been steadily developing over the previous few decades. Many different strategies have been presented in research articles based on gesture recognition to try to create an effective system to send non-verbal natural communication information to computers, using both physical sensors and computer vision. Hyper accurate real-time systems, on the other hand, have only recently began to occupy the study field, with each adopting a range of method…
Münster School of Business（明斯特商学院）；FH Münster - University of Applied Sciences（明斯特应用科学大学）
We develop Few-Shot Learning models trained to recognize five or ten different dynamic hand gestures, respectively, which are arbitrarily interchangeable by providing the model with one, two, or five examples per hand gesture. All models were built in the Few-Shot Learning architecture of the Relation Network (RN), in which Long-Short-Term Memory cells form the backbone. The models use hand reference points extracted from RGB-video sequences of the Jester dataset which was modified to contain 190 different types of hand gestures. Result show accuracy of up to 88.8% for recognition of five and …
Hand gesture recognition has long been a hot topic in human computer interaction. Traditional camera-based hand gesture recognition systems cannot work properly under dark circumstances. In this paper, a Doppler Radar based hand gesture recognition system using convolutional neural networks is proposed. A cost-effective Doppler radar sensor with dual receiving channels at 5.8GHz is used to acquire a big database of four standard gestures. The received hand gesture signals are then processed with time-frequency analysis. Convolutional neural networks are used to classify different gestures. Exp…
We propose a new technique for recognition of dumb person hand gesture in real world environment. In this technique, the hand image containing the gesture is preprocessed and then hand region is segmented by convergent the RGB color image to L.a.b color space. Only few statistical features are used to classify the segmented image to different classes. Artificial Neural Network is trained in sequential manner using one against all. When the system gets trained, it becomes capable of recognition of each class in parallel manner. The result of proposed technique is much better than existing techn…
The Pennsylvania State University（宾夕法尼亚州立大学）；Cornell University（康奈尔大学）；Microsoft Research（微软研究院）；Google Research（谷歌研究院）
Sensors on mobile devices---accelerometers, gyroscopes, pressure meters, and GPS---invite new applications in gesture recognition, gaming, and fitness tracking. However, programming them remains challenging because human gestures captured by sensors are noisy. This paper illustrates that noisy gestures degrade training and classification accuracy for gesture recognition in state-of-the-art deterministic Hidden Markov Models (HMM). We introduce a new statistical quantization approach that mitigates these problems by (1) during training, producing gesture-specific codebooks, HMMs, and error mode…
Gesture recognition application over 802.11 ad/y waveforms is developed. Simultaneous gestures of slider-control and two-finger gesture for switching are detected based on Golay sequences of channel estimation fields of the packets.
National Institute of Advanced Industrial Science and Technology（产业技术综合研究所）
In this study, we introduce a novel variant and application of the Collaborative Representation based Classification in spectral domain for recognition of the hand gestures using the raw surface Electromyography signals. The intuitive use of spectral features are explained via circulant matrices. The proposed Spectral Collaborative Representation based Classification (SCRC) is able to recognize gestures with higher levels of accuracy for a fairly rich gesture set. The worst recognition result which is the best in the literature is obtained as 97.3\% among the four sets of the experiments for e…
Ego hand gestures can be used as an interface in AR and VR environments. While the context of an image is important for tasks like scene understanding, object recognition, image caption generation and activity recognition, it plays a minimal role in ego hand gesture recognition. An ego hand gesture used for AR and VR environments conveys the same information regardless of the background. With this idea in mind, we present our work on ego hand gesture recognition that produces embeddings from RBG images with ego hands, which are simultaneously used for ego hand segmentation and ego gesture reco…
Shahid Rajaee Teacher Training University（沙希德·拉贾伊师范大学）；University of Mohaghegh Ardabili（莫哈格赫·阿尔达比勒大学）；Blaise Pascal University（布莱斯·帕斯卡大学）
At the present time, hand gestures recognition system could be used as a more expected and useable approach for human computer interaction. Automatic hand gesture recognition system provides us a new tactic for interactive with the virtual environment. In this paper, a face and hand gesture recognition system which is able to control computer media player is offered. Hand gesture and human face are the key element to interact with the smart system. We used the face recognition scheme for viewer verification and the hand gesture recognition in mechanism of computer media player, for instance, v…
In this project, we design a real-time human-computer interaction system based on hand gesture. The whole system consists of three components: hand detection, gesture recognition and human-computer interaction (HCI) based on recognition; and realizes the robust control of mouse and keyboard events with a higher accuracy of gesture recognition. Specifically, we use the convolutional neural network (CNN) to recognize gestures and makes it attainable to identify relatively complex gestures using only one cheap monocular camera. We introduce the Kalman filter to estimate the hand position based on…
Gesture recognition is getting more and more popular due to various application possibilities in human-machine interaction. Existing multi-modal gesture recognition systems take multi-modal data as input to improve accuracy, but such methods require more modality sensors, which will greatly limit their application scenarios. Therefore we propose an end-to-end multi-task learning framework in training 2D convolutional neural networks. The framework can use the depth modality to improve accuracy during training and save costs by using only RGB modality during inference. Our framework is trained …
University of Rennes 1（雷恩第一大学）；Inserm（法国国家健康与医学研究院）
In this paper, we address the open research problem of surgical gesture recognition using motion cues from video data only. We adapt Optical flow ConvNets initially proposed by Simonyan et al.. While Simonyan uses both RGB frames and dense optical flow, we use only dense optical flow representations as input to emphasize the role of motion in surgical gesture recognition, and present it as a robust alternative to kinematic data. We also overcome one of the limitations of Optical flow ConvNets by initializing our model with cross modality pre-training. A large number of promising studies that a…
Micro-gesture Online Recognition using Learnable Query Points
School of Computer Science and Information Engineering（计算机与信息工程学院）；School of Artificial Intelligence（人工智能学院）；Hefei University of Technology (HFUT)（合肥工业大学）；Key Laboratory of Knowledge Engineering with Big Data (HFUT), Ministry of Education（大数据知识工程教育部重点实验室）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）；Zhejiang University（浙江大学）；Anhui Zhonghuitong Technology Co., Ltd.（安徽中汇通科技有限公司）
TU Munich（慕尼黑工业大学）；LMU Munich（慕尼黑大学）；University of Tuebingen（图宾根大学）；Intel Labs Europe（英特尔欧洲实验室）；Intel Deutschland GmbH（英特尔德国有限公司）
The use of hand gestures provides a natural alternative to cumbersome interface devices for Human-Computer Interaction (HCI) systems. However, real-time recognition of dynamic micro hand gestures from video streams is challenging for in-vehicle scenarios since (i) the gestures should be performed naturally without distracting the driver, (ii) micro hand gestures occur within very short time intervals at spatially constrained areas, (iii) the performed gesture should be recognized only once, and (iv) the entire architecture should be designed lightweight as it will be deployed to an embedded sy…
This paper describes a technique of real time head gesture recognition system. The method includes Gaussian mixture model (GMM) accompanied by optical flow algorithm which provided us the required information regarding head movement. The proposed model can be implemented in various control system. We are also presenting the result and implementation of both mentioned method.
This paper addresses the problem of continuous gesture recognition from sequences of depth maps using convolutional neutral networks (ConvNets). The proposed method first segments individual gestures from a depth sequence based on quantity of movement (QOM). For each segmented gesture, an Improved Depth Motion Map (IDMM), which converts the depth sequence into one image, is constructed and fed to a ConvNet for recognition. The IDMM effectively encodes both spatial and temporal information and allows the fine-tuning with existing ConvNet models for classification without introducing millions of…
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）
Hand gesture recognition plays a significant role in human-computer interaction for understanding various human gestures and their intent. However, most prior works can only recognize gestures of limited labeled classes and fail to adapt to new categories. The task of Generalized Zero-Shot Learning (GZSL) for hand gesture recognition aims to address the above issue by leveraging semantic representations and detecting both seen and unseen class samples. In this paper, we propose an end-to-end prototype-based GZSL framework for hand gesture recognition which consists of two branches. The first b…
We present an on-device real-time hand gesture recognition (HGR) system, which detects a set of predefined static gestures from a single RGB camera. The system consists of two parts: a hand skeleton tracker and a gesture classifier. We use MediaPipe Hands as the basis of the hand skeleton tracker, improve the keypoint accuracy, and add the estimation of 3D keypoints in a world metric space. We create two different gesture classifiers, one based on heuristics and the other using neural networks (NN).
This paper presents the maneuver of mouse pointer and performs various mouse operations such as left click, right click, double click, drag etc using gestures recognition technique. Recognizing gestures is a complex task which involves many aspects such as motion modeling, motion analysis, pattern recognition and machine learning. Keeping all the essential factors in mind a system has been created which recognizes the movement of fingers and various patterns formed by them. Color caps have been used for fingers to distinguish it from the background color such as skin color. Thus recognizing th…
The recent introduction of depth cameras like Leap Motion Controller allows researchers to exploit the depth information to recognize hand gesture more robustly. This paper proposes a novel hand gesture recognition system with Leap Motion Controller. A series of features are extracted from Leap Motion tracking data, we feed these features along with HOG feature extracted from sensor images into a multi-class SVM classifier to recognize performed gesture, dimension reduction and feature weighted fusion are also discussed. Our results show that our model is much more accurate than previous work.
Beihang University（北京航空航天大学）；China University of Mining and Technology（中国矿业大学）；University of Central Florida（中佛罗里达大学）；Nortumbria Univesity（诺森比亚大学）
Gesture recognition is a challenging problem in the field of biometrics. In this paper, we integrate Fisher criterion into Bidirectional Long-Short Term Memory (BLSTM) network and Bidirectional Gated Recurrent Unit (BGRU),thus leading to two new deep models termed as F-BLSTM and F-BGRU. BothFisher discriminative deep models can effectively classify the gesture based on analyzing the acceleration and angular velocity data of the human gestures. Moreover, we collect a large Mobile Gesture Database (MGD) based on the accelerations and angular velocities containing 5547 sequences of 12 gestures. E…
We propose a bidirectional consecutively connected two-pathway network (BCCN) for efficient gesture recognition. The BCCN consists of two pathways: (i) a keyframe pathway and (ii) a temporal-attention pathway. The keyframe pathway is configured using the skeleton-based keyframe selection module. Keyframes pass through the pathway to extract the spatial feature of itself, and the temporal-attention pathway extracts temporal semantics. Our model improved gesture recognition performance in videos and obtained better activation maps for spatial and temporal properties. Tests were performed on the …
Mobile virtual reality (VR) head mounted displays (HMD) have become popular among consumers in recent years. In this work, we demonstrate real-time egocentric hand gesture detection and localization on mobile HMDs. Our main contributions are: 1) A novel mixed-reality data collection tool to automatic annotate bounding boxes and gesture labels; 2) The largest-to-date egocentric hand gesture and bounding box dataset with more than 400,000 annotated frames; 3) A neural network that runs real time on modern mobile CPUs, and achieves higher than 76% precision on gesture recognition across 8 classes…
In this paper we have presented a hand gesture recognition library. Various functions include detecting cluster count, cluster orientation, finger pointing direction, etc. To use these functions first the input image needs to be processed into a logical array for which a function has been developed. The library has been developed keeping flexibility in mind and thus provides application developers a wide range of options to develop custom gestures.
Gesture recognition is a much studied research area which has myriad real-world applications including robotics and human-machine interaction. Current gesture recognition methods have focused on recognising isolated gestures, and existing continuous gesture recognition methods are limited to two-stage approaches where independent models are required for detection and classification, with the performance of the latter being constrained by detection performance. In contrast, we introduce a single-stage continuous gesture recognition framework, called Temporal Multi-Modal Fusion (TMMF), that can …
In this report, an automated bartender system was developed for making orders in a bar using hand gestures. The gesture recognition of the system was developed using Machine Learning techniques, where the model was trained to classify gestures using collected data. The final model used in the system reached an average accuracy of 95%. The system raised ethical concerns both in terms of user interaction and having such a system in a real world scenario, but it could initially work as a complement to a real bartender.
Instituto Nacional de Astrofísica, Óptica y Electrónica（国家天体物理、光学与电子研究所）；University of Texas at Arlington（德克萨斯大学阿灵顿分校）；Beijing Jiaotong University（北京交通大学）
This paper introduces principal motion components (PMC), a new method for one-shot gesture recognition. In the considered scenario a single training-video is available for each gesture to be recognized, which limits the application of traditional techniques (e.g., HMMs). In PMC, a 2D map of motion energy is obtained per each pair of consecutive frames in a video. Motion maps associated to a video are processed to obtain a PCA model, which is used for recognition under a reconstruction-error approach. The main benefits of the proposed approach are its simplicity, easiness of implementation, com…
University of Trento（特伦托大学）；Peking University（北京大学）；Lingxi Artificial Intelligence Co., Ltd（灵汐人工智能有限公司）
Gesture recognition is a hot topic in computer vision and pattern recognition, which plays a vitally important role in natural human-computer interface. Although great progress has been made recently, fast and robust hand gesture recognition remains an open problem, since the existing methods have not well balanced the performance and the efficiency simultaneously. To bridge it, this work combines image entropy and density clustering to exploit the key frames from hand gesture video for further feature extraction, which can improve the efficiency of recognition. Moreover, a feature fusion stra…
Shahid Rajaee Teacher Training University（沙希德·拉贾伊师范大学）；Shahid Bahonar University（沙希德·巴哈尔大学）；Hamedan Science and Research University（哈马丹科学与研究大学）
Hand gesture recognition possesses extensive applications in virtual reality, sign language recognition, and computer games. The direct interface of hand gestures provides us a new way for communicating with the virtual environment. In this paper a novel and real-time approach for hand gesture recognition system is presented. In the suggested method, first, the hand gesture is extracted from the main image by the image segmentation and morphological operation and then is sent to feature extraction stage. In feature extraction stage the Cross-correlation coefficient is applied on the gesture to…
In this paper, it is introduced a hand gesture recognition system to recognize the characters in the real time. The system consists of three modules: real time hand tracking, training gesture and gesture recognition using Convolutional Neural Networks. Camshift algorithm and hand blobs analysis for hand tracking are being used to obtain motion descriptors and hand region. It is fairy robust to background cluster and uses skin color for hand gesture tracking and recognition. Furthermore, the techniques have been proposed to improve the performance of the recognition and the accuracy using the a…
Hand Gesture Recognition (HGR) is of major importance for Human-Computer Interaction (HCI) applications. In this paper, we present a new hand gesture recognition approach called GNG-IEMD. In this approach, first, we use a Growing Neural Gas (GNG) graph to model the image. Then we extract features from this graph. These features are not geometric or pixel-based, so do not depend on scale, rotation, and articulation. The dissimilarity between hand gestures is measured with a novel Improved Earth Mover\textquotesingle s Distance (IEMD) metric. We evaluate the performance of the proposed approach …
University of South Australia（南澳大学）；Defence Science and Technology Group（国防科学技术集团）
Current UAV-recorded datasets are mostly limited to action recognition and object tracking, whereas the gesture signals datasets were mostly recorded in indoor spaces. Currently, there is no outdoor recorded public video dataset for UAV commanding signals. Gesture signals can be effectively used with UAVs by leveraging the UAVs visual sensors and operational simplicity. To fill this gap and enable research in wider application areas, we present a UAV gesture signals dataset recorded in an outdoor setting. We selected 13 gestures suitable for basic UAV navigation and command from general aircra…
Dynamic hand gesture recognition has attracted increasing interests because of its importance for human computer interaction. In this paper, we propose a new motion feature augmented recurrent neural network for skeleton-based dynamic hand gesture recognition. Finger motion features are extracted to describe finger movements and global motion features are utilized to represent the global movement of hand skeleton. These motion features are then fed into a bidirectional recurrent neural network (RNN) along with the skeleton sequence, which can augment the motion features for RNN and improve the…
Nowadays actions are increasingly being handled in electronic ways, instead of physical interaction. From earlier times biometrics is used in the authentication of a person. It recognizes a person by using a human trait associated with it like eyes (by calculating the distance between the eyes) and using hand gestures, fingerprint detection, face detection etc. Advantages of using these traits for identification are that they uniquely identify a person and cannot be forgotten or lost. These are unique features of a human being which are being used widely to make the human life simpler. Hand ge…
Human activity and gesture recognition is an important component of rapidly growing domain of ambient intelligence, in particular in assisting living and smart homes. In this paper, we propose to combine the power of two deep learning techniques, the convolutional neural networks (CNN) and the recurrent neural networks (RNN), for automated hand gesture recognition using both depth and skeleton data. Each of these types of data can be used separately to train neural networks to recognize hand gestures. While RNN were reported previously to perform well in recognition of sequences of movement fo…
University of Tartu（塔尔图大学）；Lodz University of Technology（罗兹理工大学）；University of Barcelona（巴塞罗那大学）；Computer Vision Center（计算机视觉中心）；Hasan Kalyoncu University（哈桑·卡利永丘大学）
Automatic emotion recognition has become a trending research topic in the past decade. While works based on facial expressions or speech abound, recognizing affect from body gestures remains a less explored topic. We present a new comprehensive survey hoping to boost research in the field. We first introduce emotional body gestures as a component of what is commonly known as "body language" and comment general aspects as gender differences and culture dependence. We then define a complete framework for automatic emotional body gesture recognition. We introduce person detection and comment stat…
Manipal Institute of Technology, MAHE（玛尼帕尔高等教育学院马尼帕尔理工学院）
Using physical interactive devices like mouse and keyboards hinders naturalistic human-machine interaction and increases the probability of surface contact during a pandemic. Existing gesture-recognition systems do not possess user authentication, making them unreliable. Static gestures in current gesture-recognition technology introduce long adaptation periods and reduce user compatibility. Our technology places a strong emphasis on user recognition and safety. We use meaningful and relevant gestures for task operation, resulting in a better user experience. This paper aims to design a robust…
A car driver knows how to react on the gestures of the traffic officers. Clearly, this is not the case for the autonomous vehicle, unless it has road traffic control gesture recognition functionalities. In this work, we address the limitation of the existing autonomous driving datasets to provide learning data for traffic control gesture recognition. We introduce a dataset that is based on 3D body skeleton input to perform traffic control gesture classification on every time step. Our dataset consists of 250 sequences from several actors, ranging from 16 to 90 seconds per sequence. To evaluate…
Augmenting human computer interaction with automated analysis and synthesis of facial expressions is a goal towards which much research effort has been devoted recently. Facial gesture recognition is one of the important component of natural human-machine interfaces; it may also be used in behavioural science, security systems and in clinical practice. Although humans recognise facial expressions virtually without effort or delay, reliable expression recognition by machine is still a challenge. The face expression recognition problem is challenging because different individuals display the sam…
Natural interaction with virtual objects in AR/VR environments makes for a smooth user experience. Gestures are a natural extension from real world to augmented space to achieve these interactions. Finding discriminating spatio-temporal features relevant to gestures and hands in ego-view is the primary challenge for recognising egocentric gestures. In this work we propose a data driven end-to-end deep learning approach to address the problem of egocentric gesture recognition, which combines an ego-hand encoder network to find ego-hand features, and a recurrent neural network to discern tempora…
Universitat de Barcelona（巴塞罗那大学）；Computer Vision Center（计算机视觉中心）；Parc Taulí Fundation（塔乌利公园基金会）
We present an application of gesture recognition using an extension of Dynamic Time Warping (DTW) to recognize behavioural patterns of Attention Deficit Hyperactivity Disorder (ADHD). We propose an extension of DTW using one-class classifiers in order to be able to encode the variability of a gesture category, and thus, perform an alignment between a gesture sample and a gesture class. We model the set of gesture samples of a certain gesture category using either GMMs or an approximation of Convex Hulls. Thus, we add a theoretical contribution to classical warping path in DTW by including loca…
