Movement control of artificial limbs has made big advances in recent years. New sensor and control technology enhanced the functionality and usefulness of artificial limbs to the point that complex movements, such as grasping, can be performed to a limited extent. To date, the most successful results were achieved by applying recurrent neural networks (RNNs). However, in the domain of artificial hands, experiments so far were limited to non-mobile wrists, which significantly reduces the functionality of such prostheses. In this paper, for the first time, we present empirical results on gesture recognition with both mobile and non-mobile wrists. Furthermore, we demonstrate that recurrent neural networks with simple recurrent units (SRU) outperform regular RNNs in both cases in terms of gesture recognition accuracy, on data acquired by an arm band sensing electromagnetic signals from arm muscles (via surface electromyography or sEMG). Finally, we show that adding domain adaptation techniques to continuous gesture recognition with RNN improves the transfer ability between subjects, where a limb controller trained on data from one person is used for another person.
Recent advancements in diagnostic learning and development of gesture-based human machine interfaces have driven surface electromyography (sEMG) towards significant importance. Analysis of hand gestures requires an accurate assessment of sEMG signals. The proposed work presents a novel sequential master-slave architecture consisting of deep neural networks (DNNs) for classification of signs from the Indian sign language using signals recorded from multiple sEMG channels. The performance of the master-slave network is augmented by leveraging additional synthetic feature data generated by long s…
Surface Electromyography (sEMG/EMG) is to record muscles' electrical activity from a restricted area of the skin by using electrodes. The sEMG-based gesture recognition is extremely sensitive of inter-session and inter-subject variances. We propose a model and a deep-learning-based domain adaptation method to approximate the domain shift for recognition accuracy enhancement. Analysis performed on sparse and HighDensity (HD) sEMG public datasets validate that our approach outperforms state-of-the-art methods.
We propose a new metric to measure domain divergence and a new domain adaptation method for time-series classification. The metric belongs to the class of probability distributions-based metrics, is transductive, and does not assume the presence of source data samples. The 2-stage method utilizes an improved autoregressive, RNN-based architecture with deep/non-linear transformation. We assess our metric and the performance of our model in the context of sEMG/EMG-based gesture recognition under inter-session and inter-subject domain shifts.
A Deep Learning Sequential Decoder for Transient High-Density Electromyography in Hand Gesture Recognition Using Subject-Embedded Transfer Learning
University of California Los Angeles (UCLA)（加州大学洛杉矶分校（UCLA））；New York University (NYU)（纽约大学（NYU））；NYU WIRELESS（纽约大学无线研究中心）；NYU Center for Urban Science and Progress (CUSP)（纽约大学城市科学与进步中心（CUSP））
In this paper, we present our work on developing robot arm prosthetic via deep learning. Our work proposes to use transfer learning techniques applied to the Google Inception model to retrain the final layer for surface electromyography (sEMG) classification. Data have been collected using the Thalmic Labs Myo Armband and used to generate graph images comprised of 8 subplots per image containing sEMG data captured from 40 data points per sensor, corresponding to the array of 8 sEMG sensors in the armband. Data captured were then classified into four categories (Fist, Thumbs Up, Open Hand, Rest…
Concordia University（康考迪亚大学）；Imperial College London（伦敦帝国学院）；New York University（纽约大学）；York University（约克大学）
This work is motivated by the recent advances in Deep Neural Networks (DNNs) and their widespread applications in human-machine interfaces. DNNs have been recently used for detecting the intended hand gesture through processing of surface electromyogram (sEMG) signals. The ultimate goal of these approaches is to realize high-performance controllers for prosthetic. However, although DNNs have shown superior accuracy than conventional methods when large amounts of data are available for training, their performance substantially decreases when data are limited. Collecting large datasets for train…
Concordia University（康考迪亚大学）；New York University（纽约大学）；University of Calgary（卡尔加里大学）
Designing efficient and labor-saving prosthetic hands requires powerful hand gesture recognition algorithms that can achieve high accuracy with limited complexity and latency. In this context, the paper proposes a compact deep learning framework referred to as the CT-HGR, which employs a vision transformer network to conduct hand gesture recognition using highdensity sEMG (HD-sEMG) signals. The attention mechanism in the proposed model identifies similarities among different data segments with a greater capacity for parallel computations and addresses the memory limitation problems while deali…
Università di Bologna（博洛尼亚大学）；Politecnico di Torino（都灵理工大学）；ETH Zurich（苏黎世联邦理工学院）；University of Modena and Reggio Emilia（摩德纳-雷焦艾米利亚大学）
Human-machine interaction is gaining traction in rehabilitation tasks, such as controlling prosthetic hands or robotic arms. Gesture recognition exploiting surface electromyographic (sEMG) signals is one of the most promising approaches, given that sEMG signal acquisition is non-invasive and is directly related to muscle contraction. However, the analysis of these signals still presents many challenges since similar gestures result in similar muscle contractions. Thus the resulting signal shapes are almost identical, leading to low classification accuracy. To tackle this challenge, complex neu…
Concordia University（康考迪亚大学）；Concordia Institute for Information System Engineering (CIISE)（康考迪亚大学信息系统工程研究所（CIISE））；Electrical and Computer Engineering (ECE), Concordia University（康考迪亚大学电气与计算机工程系（ECE））；Department of Bioengineering, Imperial College London（帝国理工学院生物工程系）；Electrical and Computer Engineering and Mechanical and Aerospace Engineering, New York University (NYU)（纽约大学（NYU）电气与计算机工程及机械与航空航天工程系）；New York University (NYU)（纽约大学（NYU））；Imperial College London（帝国理工学院）
There has been a surge of recent interest in Machine Learning (ML), particularly Deep Neural Network (DNN)-based models, to decode muscle activities from surface Electromyography (sEMG) signals for myoelectric control of neurorobotic systems. DNN-based models, however, require large training sets and, typically, have high structural complexity, i.e., they depend on a large number of trainable parameters. To address these issues, we developed a framework based on the Transformer architecture for processing sEMG signals. We propose a novel Vision Transformer (ViT)-based neural network architectu…
Myoelectric control is one of the leading areas of research in the field of robotic prosthetics. We present our research in surface electromyography (sEMG) signal classification, where our simple and novel attention-based approach now leads the industry, universally beating more complex, state-of-the-art models. Our novel attention-based model achieves benchmark leading results on multiple industry-standard datasets including 53 finger, wrist, and grasping motions, improving over both sophisticated signal processing and CNN-based approaches. Our strong results with a straightforward model also…
Concordat University（康考迪亚大学）；Concordia Institute for Information System Engineering (CITIES)（康考迪亚信息系统工程研究所（CITIES））
Recently, there has been a surge of significant interest on application of Deep Learning (DL) models to autonomously perform hand gesture recognition using surface Electromyogram (sEMG) signals. DL models are, however, mainly designed to be applied on sparse sEMG signals. Furthermore, due to their complex structure, typically, we are faced with memory constraints; require large training times and a large number of training samples, and; there is the need to resort to data augmentation and/or transfer learning. In this paper, for the first time (to the best of our knowledge), we investigate and…
User Training with Error Augmentation for Electromyogram-based Gesture Classification
Northeastern University（东北大学）；Khoury College of Computer Sciences（科里计算机科学学院）；Meta Reality Labs Research（Meta现实实验室）；Institute for Experiential AI（体验式人工智能研究院）；Department of Physical Therapy, Movement, and Rehabilitation Sciences（物理治疗、运动与康复科学系）
Department of Electrical Engineering, Center of Excellence on Soft Computing and Intelligent Information Processing, Ferdowsi University of Mashhad（马什哈德费尔多西大学电气工程系，软计算与智能信息处理卓越中心）；Department of Computer Engineering, Ne. C., Islamic Azad University, Neyshabur（内沙布尔伊斯兰阿扎德大学计算机工程系）
Deep learning-based Hand Gesture Recognition (HGR) via surface Electromyogram (sEMG) signals has recently shown significant potential for development of advanced myoelectric-controlled prosthesis. Existing deep learning approaches, typically, include only one model as such can hardly maintain acceptable generalization performance in changing scenarios. In this paper, we aim to address this challenge by capitalizing on the recent advances of hybrid models and transformers. In other words, we propose a hybrid framework based on the transformer architecture, which is a relatively new and revoluti…
Surface electromyography (sEMG) provides an intuitive and non-invasive interface from which to control machines. However, preserving the myoelectric control system's performance over multiple days is challenging, due to the transient nature of the signals obtained with this recording technique. In practice, if the system is to remain usable, a time-consuming and periodic recalibration is necessary. In the case where the sEMG interface is employed every few days, the user might need to do this recalibration before every use. Thus, severely limiting the practicality of such a control method. Con…
Surface electromyography (sEMG) is becoming exceeding useful in applications involving analysis of human motion such as in human-machine interface, assistive technology, healthcare and prosthetic development. The proposed work presents a novel dual stage classification approach for classification of grasping gestures from sEMG signals. A statistical assessment of these activities is presented to determine the similar characteristics between the considered activities. Similar activities are grouped together. In the first stage of classification, an activity is identified as belonging to a group…
Deep convolutional neural networks (CNNs) are appealing for the purpose of classification of hand movements from surface electromyography (sEMG) data because they have the ability to perform automated person-specific feature extraction from raw data. In this paper, we make the novel contribution of proposing and evaluating a design for the early processing layers in the deep CNN for multichannel sEMG. Specifically, we propose a novel temporal-to-spatial (TtS) CNN architecture, where the first layer performs convolution separately on each sEMG channel to extract temporal features. This is motiv…
Surface electromyography (sEMG) has gained significant importance during recent advancements in consumer electronics for healthcare systems, gesture analysis and recognition and sign language communication. For such a system, it is imperative to determine the regions of activity in a continuously recorded sEMG signal. The proposed work provides a novel activity detection approach based on Hidden Markov Models (HMM) using sEMG signals recorded when various hand gestures are performed. Detection procedure is designed based on a probabilistic outlook by making use of mathematical models. The requ…
Concordia University（康考迪亚大学）；Concordia Institute for Information System Engineering（康考迪亚信息系统工程研究院）；New York University（纽约大学）
Capitalizing on the need for addressing the existing challenges associated with gesture recognition via sparse multichannel surface Electromyography (sEMG) signals, the paper proposes a novel deep learning model, referred to as the XceptionTime architecture. The proposed innovative XceptionTime is designed by integration of depthwise separable convolutions, adaptive average pooling, and a novel non-linear normalization technique. At the heart of the proposed architecture is several XceptionTime modules concatenated in series fashion designed to capture both temporal and spatial information-bea…
Concordia University（康考迪亚大学）；Concordia Institute for Information System Engineering（康考迪亚信息系统工程研究院）；Imperial College London（帝国理工学院）；New York University（纽约大学）
Advances in biosignal signal processing and machine learning, in particular Deep Neural Networks (DNNs), have paved the way for the development of innovative Human-Machine Interfaces for decoding the human intent and controlling artificial limbs. DNN models have shown promising results with respect to other algorithms for decoding muscle electrical activity, especially for recognition of hand gestures. Such data-driven models, however, have been challenged by their need for a large number of trainable parameters and their structural complexity. Here we propose the novel Temporal Convolutions-b…
Concordia University（康考迪亚大学）；Concordia Institute for Information System Engineering（康考迪亚信息系统工程研究所）
Advancements in Biological Signal Processing (BSP) and Machine-Learning (ML) models have paved the path for development of novel immersive Human-Machine Interfaces (HMI). In this context, there has been a surge of significant interest in Hand Gesture Recognition (HGR) utilizing Surface-Electromyogram (sEMG) signals. This is due to its unique potential for decoding wearable data to interpret human intent for immersion in Mixed Reality (MR) environments. To achieve the highest possible accuracy, complicated and heavy-weighted Deep Neural Networks (DNNs) are typically developed, which restricts t…
Bias in Surface Electromyography Features across a Demographically Diverse Cohort
Graduate Group in Computer Science, University of California, Davis（计算机科学研究生组，加州大学戴维斯分校）；Department of Electrical and Computer Engineering, University of California, Davis（电气与计算机工程系，加州大学戴维斯分校）；Department of Mechanical and Mechatronic Engineering and Advanced Manufacturing, California State University, Chico（机械与机电工程及先进制造系，加州州立大学奇科分校）；Department of Neurobiology, Physiology & Behavior, University of California, Davis（神经生物学、生理学与行为系，加州大学戴维斯分校）；Department of Otolaryngology / Head & Neck Surgery, University of California, Davis（耳鼻喉科/头颈外科系，加州大学戴维斯分校）；Center for Mind & Brain, University of California, Davis（心智与大脑中心，加州大学戴维斯分校）；Department of Mechanical & Aerospace Engineering, University of California, Davis（机械与航空航天工程系，加州大学戴维斯分校）
Multimodal Surface EMG Hand Gesture Recognition Using Query-Based Transformers for Prosthetic Control
Department of Information Engineering, University of Padua（帕多瓦大学信息工程系）；Padova Neuroscience Center, University of Padua（帕多瓦大学帕多瓦神经科学中心）；Department of Biomedical Sciences, University of Padua（帕多瓦大学生物医学科学系）；Department of Neuroscience, University of Padua（帕多瓦大学神经科学系）；Information Systems Institute, University of Applied Sciences Western Switzerland (HES-SO Valais)（瑞士西部应用科学大学（HES-SO瓦莱州）信息系统研究所）
Concordia University（康考迪亚大学）；Concordia Institute of Information Systems Engineering (CIISE)（康考迪亚信息系统工程研究院）；New York University (NYU)（纽约大学）；UNSW SYDNEY（悉尼新南威尔士大学）
Development of advance surface Electromyogram (sEMG)-based Human-Machine Interface (HMI) systems is of paramount importance to pave the way towards emergence of futuristic Cyber-Physical-Human (CPH) worlds. In this context, the main focus of recent literature was on development of different Deep Neural Network (DNN)-based architectures that perform Hand Gesture Recognition (HGR) at a macroscopic level (i.e., directly from sEMG signals). At the same time, advancements in acquisition of High-Density sEMG signals (HD-sEMG) have resulted in a surge of significant interest on sEMG decomposition tec…
Gesture recognition is a very essential technology for many wearable devices. While previous algorithms are mostly based on statistical methods including the hidden Markov model, we develop two dynamic hand gesture recognition techniques using low complexity recurrent neural network (RNN) algorithms. One is based on video signal and employs a combined structure of a convolutional neural network (CNN) and an RNN. The other uses accelerometer data and only requires an RNN. Fixed-point optimization that quantizes most of the weights into two bits is conducted to optimize the amount of memory size…
Univ de Lyon（里昂大学）；INSA-Lyon（里昂国立应用科学学院）；CNRS（法国国家科学研究中心）；LIRIS（里昂图像与信息系统实验室）；INRIA（法国国家信息与自动化研究所）；CITI（信息与交互技术中心）；Normandie Univ（诺曼底大学）；INSA Rouen（鲁昂国立应用科学学院）；UNIROUEN（鲁昂大学）；UNIHAVRE（勒阿弗尔大学）；LITIS（信息处理与智能系统实验室）；Itekube（Itekube公司）
We propose a fully automatic method for learning gestures on big touch devices in a potentially multi-user context. The goal is to learn general models capable of adapting to different gestures, user styles and hardware variations (e.g. device sizes, sampling frequencies and regularities). Based on deep neural networks, our method features a novel dynamic sampling and temporal normalization component, transforming variable length gestures into fixed length representations while preserving finger/surface contact transitions, that is, the topology of the signal. This sequential representation is…
Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS)（惠康/工程与物理科学研究委员会介入与外科科学中心（WEISS））；University College London（伦敦大学学院）
Surgical gesture recognition is important for surgical data science and computer-aided intervention. Even with robotic kinematic information, automatically segmenting surgical steps presents numerous challenges because surgical demonstrations are characterized by high variability in style, duration and order of actions. In order to extract discriminative features from the kinematic signals and boost recognition accuracy, we propose a multi-task recurrent neural network for simultaneous recognition of surgical gestures and estimation of a novel formulation of surgical task progress. To show the…
By using a computer keyboard as a finger recording device, we construct the largest existing dataset for gesture recognition via surface electromyography (sEMG), and use deep learning to achieve over 90% character-level accuracy on reconstructing typed text entirely from measured muscle potentials. We prioritize the temporal structure of the EMG signal instead of the spatial structure of the electrode layout, using network architectures inspired by those used for real-time spoken language transcription. Our architecture recognizes the rapid movements of natural computer typing, which occur at …
The use of sensors has pervaded everyday life in several applications including human activity monitoring, healthcare, and social networks. In this study, we focus on the use of smartwatch sensors to recognize smoking activity. More specifically, we have reformulated the previous work in detection of smoking to include in-context recognition of smoking. Our presented reformulation of the smoking gesture as a state-transition model that consists of the mini-gestures hand-to-lip, hand-on-lip, and hand-off-lip, has demonstrated improvement in detection rates nearing 100% using conventional neural…
Resonate-and-Fire Spiking Neurons for Target Detection and Hand Gesture Recognition: A Hybrid Approach
University of Erlangen-Nuremberg（埃尔朗根-纽伦堡大学）；Infineon Technologies AG（英飞凌科技股份公司）；Institute for Electronics Engineering（电子工程研究所）；Otto-von-Guericke-University Magdeburg（马格德堡奥托·冯·格里克大学）
KTH Royal Institute of Technology（瑞典皇家理工学院）；Watty AB；Purdue University（普渡大学）
This paper introduces Wisture, a new online machine learning solution for recognizing touch-less dynamic hand gestures on a smartphone. Wisture relies on the standard Wi-Fi Received Signal Strength (RSS) using a Long Short-Term Memory (LSTM) Recurrent Neural Network (RNN), thresholding filters and traffic induction. Unlike other Wi-Fi based gesture recognition methods, the proposed method does not require a modification of the smartphone hardware or the operating system, and performs the gesture recognition without interfering with the normal operation of other smartphone applications. We disc…
Recent studies have demonstrated the power of recurrent neural networks for machine translation, image captioning and speech recognition. For the task of capturing temporal structure in video, however, there still remain numerous open research questions. Current research suggests using a simple temporal feature pooling strategy to take into account the temporal aspect of video. We demonstrate that this method is not sufficient for gesture recognition, where temporal information is more discriminative compared to general video classification tasks. We explore deep architectures for gesture reco…
Unsupervised clustering and classification of upper limb EMG signals during functional movements: a data-driven
In2Lab, Engineer Faculty, Universidad de Antioquia（1 In2实验室，工程师学院，安提奥基亚大学）；School of Engineering and Sciencies, Tecnológico de Monterrey（2 工程与科学学院，蒙特雷技术学院）
North University of China（中北大学）；Shandong University（山东大学）；University of Electronic Science and Technology of China（电子科技大学）；University of Wollongong（伍伦贡大学）
Pose based hand gesture recognition has been widely studied in the recent years. Compared with full body action recognition, hand gesture involves joints that are more spatially closely distributed with stronger collaboration. This nature requires a different approach from action recognition to capturing the complex spatial features. Many gesture categories, such as "Grab" and "Pinch", have very similar motion or temporal patterns posing a challenge on temporal processing. To address these challenges, this paper proposes a two-stream neural network with one stream being a self-attention based …
Hand Gesture Recognition from Doppler Radar Signals Using Echo State Networks
Department of Computer Science, Nagoya Institute of Technology（名古屋技术大学计算机科学系）；International Research Center for Neurointelligence, The University of Tokyo（东京大学神经智能国际研究中心）
CWI（CWI研究所）；Stichting IMEC Netherlands, Holst Centre（荷兰IMEC基金会荷尔斯特中心）；Univ of Amsterdam（阿姆斯特丹大学）；Rijksuniversiteit Groningen（格罗宁根大学）
Inspired by more detailed modeling of biological neurons, Spiking neural networks (SNNs) have been investigated both as more biologically plausible and potentially more powerful models of neural computation, and also with the aim of extracting biological neurons' energy efficiency; the performance of such networks however has remained lacking compared to classical artificial neural networks (ANNs). Here, we demonstrate how a novel surrogate gradient combined with recurrent networks of tunable and adaptive spiking neurons yields state-of-the-art for SNNs on challenging benchmarks in the time-do…
