Classifying limb movements using brain activity is an important task in Brain-computer Interfaces (BCI) that has been successfully used in multiple application domains, ranging from human-computer interaction to medical and biomedical applications. This paper proposes a novel solution for classification of left/right hand movement by exploiting a Long Short-Term Memory (LSTM) network with attention mechanism to learn the electroencephalogram (EEG) time-series information. To this end, a wide range of time and frequency domain features are extracted from the EEG signals and used to train an LSTM network to perform the classification task. We conduct extensive experiments with the EEG Movement dataset and show that our proposed solution our method achieves improvements over several benchmarks and state-of-the-art methods in both intra-subject and cross-subject validation schemes. Moreover, we utilize the proposed framework to analyze the information as received by the sensors and monitor the activated regions of the brain by tracking EEG topography throughout the experiments.
The University of Western Australia（西澳大学）；University of Jeddah（吉达大学）
Seizure type identification is essential for the treatment and management of epileptic patients. However, it is a difficult process known to be time consuming and labor intensive. Automated diagnosis systems, with the advancement of machine learning algorithms, have the potential to accelerate the classification process, alert patients, and support physicians in making quick and accurate decisions. In this paper, we present a novel multi-path seizure-type classification deep learning network (MP-SeizNet), consisting of a convolutional neural network (CNN) and a bidirectional long short-term me…
K. N. Toosi University of Technology（K. N. 图西理工大学）；Islamic Azad University（伊斯兰阿扎德大学）；University of La Rioja（拉里奥哈大学）；Deakin University（迪肯大学）；Macquarie University（麦考瑞大学）；Harvard University（哈佛大学）；University of Leicester（莱斯特大学）；Universidad de Granada（格拉纳达大学）；University of Cambridge（剑桥大学）
Schizophrenia (SZ) is a mental disorder whereby due to the secretion of specific chemicals in the brain, the function of some brain regions is out of balance, leading to the lack of coordination between thoughts, actions, and emotions. This study provides various intelligent deep learning (DL)-based methods for automated SZ diagnosis via electroencephalography (EEG) signals. The obtained results are compared with those of conventional intelligent methods. To implement the proposed methods, the dataset of the Institute of Psychiatry and Neurology in Warsaw, Poland, has been used. First, EEG sig…
Hunan University（湖南大学）；University of Warwick（华威大学）；University at Buffalo（纽约州立大学布法罗分校）；IBM Research AI（IBM研究院人工智能实验室）
The "mind-controlling" capability has always been in mankind's fantasy. With the recent advancements of electroencephalograph (EEG) techniques, brain-computer interface (BCI) researchers have explored various solutions to allow individuals to perform various tasks using their minds. However, the commercial off-the-shelf devices to run accurate EGG signal collection are usually expensive and the comparably cheaper devices can only present coarse results, which prevents the practical application of these devices in domestic services. To tackle this challenge, we propose and develop an end-to-end…
Recording Brain Activity While Listening to Music Using Wearable EEG Devices Combined with Bidirectional Long Short-Term Memory Networks
School of Music, Jiangxi Normal University（江西师范大学音乐学院）；School of Electronic Information, HuZhou College（湖州学院电子信息学院）；San Francisco State University（旧金山州立大学）
Enhancing Robustness of Asynchronous EEG-Based Movement Prediction using Classifier Ensembles
Robotics Innovation Center（机器人创新中心）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）；Institute of Medical Technology Systems（医疗技术系统研究所）；University of Duisburg-Essen（杜伊斯堡- Essen大学）
Spatiotemporal EEG-Based Emotion Recognition Using SAM Ratings from Serious Games with Hybrid Deep Learning
Department of Computer Science, Electrical Engineering and Mathematical Sciences, Western Norway University of Applied Sciences（计算机科学、电气工程与数学科学系，西挪威应用科学大学）
School of Engineering and Applied Science, Aston University（阿斯顿大学工程与应用科学学院）
This study suggests a new approach to EEG data classification by exploring the idea of using evolutionary computation to both select useful discriminative EEG features and optimise the topology of Artificial Neural Networks. An evolutionary algorithm is applied to select the most informative features from an initial set of 2550 EEG statistical features. Optimisation of a Multilayer Perceptron (MLP) is performed with an evolutionary approach before classification to estimate the best hyperparameters of the network. Deep learning and tuning with Long Short-Term Memory (LSTM) are also explored, a…
Electroencephalograph (EEG) emotion recognition is a significant task in the brain-computer interface field. Although many deep learning methods are proposed recently, it is still challenging to make full use of the information contained in different domains of EEG signals. In this paper, we present a novel method, called four-dimensional attention-based neural network (4D-aNN) for EEG emotion recognition. First, raw EEG signals are transformed into 4D spatial-spectral-temporal representations. Then, the proposed 4D-aNN adopts spectral and spatial attention mechanisms to adaptively assign the …
We propose an end to end deep learning approach for generating real-time facial animation from just audio. Specifically, our deep architecture employs deep bidirectional long short-term memory network and attention mechanism to discover the latent representations of time-varying contextual information within the speech and recognize the significance of different information contributed to certain face status. Therefore, our model is able to drive different levels of facial movements at inference and automatically keep up with the corresponding pitch and latent speaking style in the input audio…
Speech-related Brain Computer Interfaces (BCI) aim primarily at finding an alternative vocal communication pathway for people with speaking disabilities. As a step towards full decoding of imagined speech from active thoughts, we present a BCI system for subject-independent classification of phonological categories exploiting a novel deep learning based hierarchical feature extraction scheme. To better capture the complex representation of high-dimensional electroencephalography (EEG) data, we compute the joint variability of EEG electrodes into a channel cross-covariance matrix. We then extra…
University of Padova（帕多瓦大学）；National, Inter-University Consortium for Telecommunications (CNIT)（全国大学校际电信联盟（CNIT））；National Center of Neurology and Psychiatry（国立精神神经医疗研究中心）；University of Milano-Bicocca（米兰比可卡大学）；Wroclaw University of Science and Technology（弗罗茨瓦夫理工大学）
Objective: To evaluate the impact on Electroencephalography (EEG) classification of different kinds of attention mechanisms in Deep Learning (DL) models. Methods: We compared three attention-enhanced DL models, the brand-new InstaGATs, an LSTM with attention and a CNN with attention. We used these models to classify normal and abnormal (i.e., artifactual or pathological) EEG patterns. Results: We achieved the state of the art in all classification problems, regardless the large variability of the datasets and the simple architecture of the attention-enhanced models. We could also prove that, d…
School of Electrical and Computer Engineering, University of Tehran（德黑兰大学电气与计算机工程学院）；Division of Intelligent Future Technologies, Malardalen University（马拉达伦大学智能未来技术系）
Long Short-Term Memory (LSTM) is widely used in various sequential applications. Complex LSTMs could be hardly deployed on wearable and resourced-limited devices due to the huge amount of computations and memory requirements. Binary LSTMs are introduced to cope with this problem, however, they lead to significant accuracy loss in some application such as EEG classification which is essential to be deployed in wearable devices. In this paper, we propose an efficient multi-level binarized LSTM which has significantly reduced computations whereas ensuring an accuracy pretty close to full precisio…
Bharati Vidyapeeth’s College of Engineering（巴拉蒂维迪亚佩特工程学院）
The electroencephalogram (EEG) is one of the most precious technologies to understand the happenings inside our brain and further understand our body's happenings. Automatic prediction of oncoming seizures using the EEG signals helps the doctors and clinical experts and reduces their workload. This paper proposes an end-to-end deep learning algorithm to fully automate seizure prediction's laborious task without any heavy pre-processing on the EEG data or feature engineering. The proposed deep learning network is a blend of signal processing and deep learning pipeline, which automates the seizu…
For EEG-based drowsiness recognition, it is desirable to use subject-independent recognition since conducting calibration on each subject is time-consuming. In this paper, we propose a novel Convolutional Neural Network (CNN)-Long Short-Term Memory (LSTM) model for subject-independent drowsiness recognition from single-channel EEG signals. Different from existing deep learning models that are mostly treated as black-box classifiers, the proposed model can explain its decisions for each input sample by revealing which parts of the sample contain important features identified by the model for cl…
School of Electrical and Computer Engineering, University of Tehran（德黑兰大学电气与计算机工程学院）；Division of Intelligent Future Technologies, Malardalen University（马拉达伦大学智能未来技术系）
Recurrent Neural Networks (RNN) are widely used for learning sequences in applications such as EEG classification. Complex RNNs could be hardly deployed on wearable devices due to their computation and memory-intensive processing patterns. Generally, reduction in precision leads much more efficiency and binarized RNNs are introduced as energy-efficient solutions. However, naive binarization methods lead to significant accuracy loss in EEG classification. In this paper, we propose a multi-level binarized LSTM, which significantly reduces computations whereas ensuring an accuracy pretty close to…
EEG classifier cross-task transfer to avoid training sessions in robot-assisted rehabilitation
Robotics Innovation Center, German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心机器人创新中心）；Institute of Medical Technology Systems, University of Duisburg-Essen（杜伊斯堡-埃森大学医学技术系统研究所）
We present a new dataset and benchmark with the goal of advancing research in the intersection of brain activities and eye movements. Our dataset, EEGEyeNet, consists of simultaneous Electroencephalography (EEG) and Eye-tracking (ET) recordings from 356 different subjects collected from three different experimental paradigms. Using this dataset, we also propose a benchmark to evaluate gaze prediction from EEG measurements. The benchmark consists of three tasks with an increasing level of difficulty: left-right, angle-amplitude and absolute position. We run extensive experiments on this benchma…
Upper limb movement classification, which maps input signals to the target activities, is a key building block in the control of rehabilitative robotics. Classifiers are trained for the rehabilitative system to comprehend the desires of the patient whose upper limbs do not function properly. Electromyography (EMG) signals and Electroencephalography (EEG) signals are used widely for upper limb movement classification. By analysing the classification results of the real-time EEG and EMG signals, the system can understand the intention of the user and predict the events that one would like to car…
University of Minnesota（明尼苏达大学）；City University of Hong Kong（香港城市大学）；Sun Yat-Sen University（中山大学）
There is increasing interest in using deep learning approach for EEG analysis as there are still rooms for the improvement of EEG analysis in its accuracy. Convolutional long short-term (CNNLSTM) has been successfully applied in time series data with spatial structure through end-to-end learning. Here, we proposed a CNNLSTM based neural network architecture termed EEG_CNNLSTMNet for the classification of EEG signals in response to grating stimuli with different spatial frequencies. EEG_CNNLSTMNet comprises two convolutional layers and one bidirectional long short-term memory (LSTM) layer. The …
Human brain performs remarkably well in segregating a particular speaker from interfering ones in a multi-speaker scenario. It has been recently shown that we can quantitatively evaluate the segregation capability by modelling the relationship between the speech signals present in an auditory scene and the cortical signals of the listener measured using electroencephalography (EEG). This has opened up avenues to integrate neuro-feedback into hearing aids whereby the device can infer user's attention and enhance the attended speaker. Commonly used algorithms to infer the auditory attention are …
University of Engineering and Technology, Taxila（塔克西拉工程技术大学）；Children’s National Hospital（国家儿童医院）；Sheikh Zayed Institute for Pediatric Surgical Innovation（谢赫·扎耶德儿科外科创新研究所）；George Washington University（乔治华盛顿大学）
The loss of limb motion arising from damage to the spinal cord is a disability that could effect people while performing their day-to-day activities. The restoration of limb movement would enable people with spinal cord injury to interact with their environment more naturally and this is where a brain-computer interface (BCI) system could be beneficial. The detection of limb movement imagination (MI) could be significant for such a BCI, where the detected MI can guide the computer system. Using MI detection through electroencephalography (EEG), we can recognize the imagination of movement in a…
Brain-Computer Interface (BCI) is an essential mechanism that interprets the human brain signal. It provides an assistive technology that enables persons with motor disabilities to communicate with the world and also empowers them to lead independent lives. The common BCI devices use Electroencephalography (EEG) electrical activity recorded from the scalp. EEG signals are noisy owing to the presence of many artifacts, namely, eye blink, head movement, and jaw movement. Such artifacts corrupt the EEG signal and make EEG analysis challenging. This issue is addressed by locating the artifacts and…
The Brain-Computer Interface system is a profoundly developing area of experimentation for Motor activities which plays vital role in decoding cognitive activities. Classification of Cognitive-Motor Imagery activities from EEG signals is a critical task. Hence proposed a unique algorithm for classifying left/right-hand movements by utilizing Multi-layer Perceptron Neural Network. Handcrafted statistical Time domain and Power spectral density frequency domain features were extracted and obtained a combined accuracy of 96.02%. Results were compared with the deep learning framework. In addition t…
University of Padova（帕多瓦大学）；Graz University of Technology（格拉茨技术大学）；National Center of Neurology and Psychiatry（国立精神神经医疗研究中心）
The classification of different fine hand movements from EEG signals represents a relevant research challenge, e.g., in brain-computer interface applications for motor rehabilitation. Here, we analyzed two different datasets where fine hand movements (touch, grasp, palmar and lateral grasp) were performed in a self-paced modality. We trained and tested a newly proposed convolutional neural network (CNN), and we compared its classification performance into respect to two well-established machine learning models, namely, a shrinked-LDA and a Random Forest. Compared to previous literature, we too…
Malaviya National Institute of Technology（马拉维亚国家理工学院）；Maharaja Surajmal Institute of Technology（马哈拉贾·苏拉杰马尔理工学院）
In the field of neuroscience, Brain activity analysis is always considered as an important area. Schizophrenia(Sz) is a brain disorder that severely affects the thinking, behaviour, and feelings of people all around the world. Electroencephalography (EEG) is proved to be an efficient biomarker in Sz detection. EEG is a non-linear time-seriesi signal and utilizing it for investigation is rather crucial due to its non-linear structure. This paper aims to improve the performance of EEG based Sz detection using a deep learning approach. A novel hybrid deep learning model known as SzHNN (Schizophre…
A Comparative Study of Conventional and Tripolar EEG for High-Performance Reach-to-Grasp BCI Systems
Department of Electrical, Computer, and Biomedical Engineering, University of Rhode Island（电气、计算机与生物医学工程系，罗德岛大学）；Department of Mechanical Engineering, University of Rhode Island（机械工程系，罗德岛大学）
ST-Topo GAN: A Motor EEG-to-EMG Decoding Model Matched to Wrist Movement Complexity
Zhengzhou University（郑州大学）；HumVerse (Zhengzhou) Technologies Co., Ltd.（HumVerse（郑州）科技有限公司）；Henan Provincial Key Laboratory of Brain Science and Brain-Computer Interface Technology（河南省脑科学与脑机接口技术重点实验室）；Xi’an Jiaotong University（西安交通大学）；University of Electronic Science and Technology of China（电子科技大学）；Chinese Academy of Medical Sciences（中国医学科学院）
We employ both random forests and LSTM networks (more precisely CuDNNLSTM) as training methodologies to analyze their effectiveness in forecasting out-of-sample directional movements of constituent stocks of the S&P 500 from January 1993 till December 2018 for intraday trading. We introduce a multi-feature setting consisting not only of the returns with respect to the closing prices, but also with respect to the opening prices and intraday returns. As trading strategy, we use Krauss et al. (2017) and Fischer & Krauss (2018) as benchmark. On each trading day, we buy the 10 stocks with the highe…
Predicting movement of objects while the action of learning agent interacts with the dynamics of the scene still remains a key challenge in robotics. We propose a multi-layer Long Short Term Memory (LSTM) autoendocer network that predicts future frames for a robot navigating in a dynamic environment with moving obstacles. The autoencoder network is composed of a state and action conditioned decoder network that reconstructs the future frames of video, conditioned on the action taken by the agent. The input image frames are first transformed into low dimensional feature vectors with a pre-train…
EEG-based AI-BCI Wheelchair Advancement: Transformer-Based Learning with Motor Imagery for Brain Computer Interface
Department of Computer Science and Engineering, Kathmandu University（加德满都大学计算机科学与工程系）；Department of Electronics and Communication Engineering, Nepal Engineering College, Pokhara University（博卡拉大学尼泊尔工程学院电子与通信工程系）；Department of Applied Sciences and Chemical Engineering, Institute of Engineering, Pulchowk Campus, Tribhuvan University（特里布万大学工程学院普尔乔克校区应用科学与化学工程系）
A brain-computer interface (BCI) provides a direct communication pathway between user and external devices. Electroencephalogram (EEG) motor imagery (MI) paradigm is widely used in non-invasive BCI to obtain encoded signals contained user intention of movement execution. However, EEG has intricate and non-stationary properties resulting in insufficient decoding performance. By imagining numerous movements of a single-arm, decoding performance can be improved without artificial command matching. In this study, we collected intuitive EEG data contained the nine different types of movements of a …
Subject-independent Classification of Meditative State from the Resting State using EEG
Dept. of Elect. and Comm. Engg. College of Engineering Trivandrum（电子与通信工程系 工程学院特里凡布尔姆）；Centre for Brain Research Indian Institute of Science, Bangalore（脑研究中心印度科学研究院，班加罗尔）；Language Technologies Institute Carnegie Mellon University（语言技术研究所卡内基梅隆大学）；School of Psychology and Education Rishihood University, Sonipat（心理学与教育学院里希大学，索尼帕特）；Dept. of Heritage Science and Technology Indian Institute of Technology, Hyderabad（遗产科学与技术系印度理工学院，海得拉巴）
Monash University（莫纳什大学）；The University of Melbourne（墨尔本大学）
Objective Kalman filtering has previously been applied to track neural model states and parameters, particularly at the scale relevant to EEG. However, this approach lacks a reliable method to determine the initial filter conditions and assumes that the distribution of states remains Gaussian. This study presents an alternative, data-driven method to track the states and parameters of neural mass models (NMMs) from EEG recordings using deep learning techniques, specifically an LSTM neural network. Approach An LSTM filter was trained on simulated EEG data generated by a neural mass model using …
EEG Right & Left Voluntary Hand Movement-based Virtual Brain-Computer Interfacing Keyboard Using Hybrid Deep Learning Approach
Nepal Engineering College（尼泊尔工程学院）；Pokhara University（博卡拉大学）；University of Delaware（特拉华大学）；Institute of Engineering Pulchowk Campus（工程学院普尔乔克校区）；Tribhuvan University（特里布文大学）；Nepal Research and Collaboration Center（尼泊尔研究与合作中心）
Baylor College of Medicine（贝勒医学院）；Michael E. DeBakey VA Medical Center（迈克尔·E·德贝基退伍军人事务医疗中心）
The Early diagnosis and treatment of depression is essential for effective treatment. Depression, while being one of the most common mental illnesses, is still poorly understood in both research and clinical practice. Among different treatments, drug prescription is widely used, however the drug treatment is not effective for many patients. In this work, we propose a method for major depressive disorder (MDD) diagnosis as well as a method for predicting the drug response in patient with MDD using EEG signals. Method: We employ transformers, which are modified recursive neural networks with nov…
Driver vigilance estimation is an important task for transportation safety. Wearable and portable brain-computer interface devices provide a powerful means for real-time monitoring of the vigilance level of drivers to help with avoiding distracted or impaired driving. In this paper, we propose a novel multimodal architecture for in-vehicle vigilance estimation from Electroencephalogram and Electrooculogram. To enable the system to focus on the most salient parts of the learned multimodal representations, we propose an architecture composed of a capsule attention mechanism following a deep Long…
Northeastern University（东北大学）；Graz University of Technology（格拉茨工业大学）；University of Pittsburgh（匹兹堡大学）
During daily activities, humans use their hands to grasp surrounding objects and perceive sensory information which are also employed for perceptual and motor goals. Multiple cortical brain regions are known to be responsible for sensory recognition, perception and motor execution during sensorimotor processing. While various research studies particularly focus on the domain of human sensorimotor control, the relation and processing between motor execution and sensory processing is not yet fully understood. Main goal of our work is to discriminate textured surfaces varying in their roughness l…
Nanyang Technological University（南洋理工大学）；Duke University - National University of Singapore Medical School（杜克大学-新加坡国立大学医学院）；Singapore General Hospital（新加坡中央医院）
In this paper, we propose a deep learning framework, TSception, for emotion detection from electroencephalogram (EEG). TSception consists of temporal and spatial convolutional layers, which learn discriminative representations in the time and channel domains simultaneously. The temporal learner consists of multi-scale 1D convolutional kernels whose lengths are related to the sampling rate of the EEG signal, which learns multiple temporal and frequency representations. The spatial learner takes advantage of the asymmetry property of emotion responses at the frontal brain area to learn the discr…
An LSTM Feature Imitation Network for Hand Movement Recognition from sEMG Signals
New York University Abu Dhabi（纽约大学阿布扎比分校）；New York University（纽约大学）；Center for AI and Robotics (CAIR)（人工智能与机器人中心）；NYUAD Research Institute（纽约大学阿布扎比分校研究院）
Deep Learning CNN and Recurrence Analysis for Alpha Gamma EEG Biomarkers in Fragile X Syndrome
School of Information Technology (SoIT), University of Cincinnati（辛辛那提大学信息技术学院）；Division of Child and Adolescent Psychiatry Cincinnati Children’s Hospital Medical Center（辛辛那提儿童医院医学中心儿童与青少年精神病学科）
CMCRD: Cross-Modal Contrastive Representation Distillation for Emotion Recognition
Key Laboratory of Image Processing and Intelligent Control, Ministry of Education, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（图像处理与智能控制重点实验室、教育部、自动化学院、华中科技大学）；Wuhan Institute of Digital Engineering（武汉数字工程研究所）；Shanghai Jiao Tong University（上海交通大学）
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；Massachusetts Institute of Technology（麻省理工学院）
This paper studies the classification problem on electroencephalogram (EEG) data of mental tasks, using standard architecture of three-layer CNN, stacked LSTM, stacked GRU. We further propose a novel classifier - a mixed LSTM model with a CNN decoder. A hyperparameter optimization on CNN shows validation accuracy of 72% and testing accuracy of 62%. The stacked LSTM and GRU models with FFT preprocessing and downsampling on data achieve 55% and 51% testing accuracy respectively. As for the mixed LSTM model with CNN decoder, validation accuracy of 75% and testing accuracy of 70% are obtained. We …
Multiple Time Series Fusion Based on LSTM An Application to CAP A Phase Classification Using EEG
Interactive Technologies Institute（交互技术研究所）；University of Madeira（马德拉大学）；NOVA Laboratory for Computer Science and Informatics（NOVA计算机科学与信息学实验室）；Faculty of Exact Sciences and Engineering, University of Madeira（马德拉大学精确科学与工程学院）；Institute for Technological Development and Innovation in Communications, Universidad de Las Palmas de Gran Canaria（拉斯帕尔马斯大学通信技术发展与创新研究所）
How Long short-term memory artificial neural network, synthetic data, and fine-tuning improve the classification of raw EEG data
Laboratory of Data Analysis；Bioinformatics, Center for Artificial Intelligence, Innopolis University Innopolis, Russia；Cognitive Technology Laboratory, Center for Technologies in Robotics；Mechatronics Components, Innopolis University Innopolis, Russia
