Measurement of stride-related, biomechanical parameters is the common rationale for objective gait impairment scoring. State-of-the-art double integration approaches to extract these parameters from inertial sensor data are, however, limited in their clinical applicability due to the underlying assumptions. To overcome this, we present a method to translate the abstract information provided by wearable sensors to context-related expert features based on deep convolutional neural networks. Regarding mobile gait analysis, this enables integration-free and data-driven extraction of a set of 8 spatio-temporal stride parameters. To this end, two modelling approaches are compared: A combined network estimating all parameters of interest and an ensemble approach that spawns less complex networks for each parameter individually. The ensemble approach is outperforming the combined modelling in the current application. On a clinically relevant and publicly available benchmark dataset, we estimate stride length, width and medio-lateral change in foot angle up to ${-0.15\pm6.09}$ cm, ${-0.09\pm4.22}$ cm and ${0.13 \pm 3.78^\circ}$ respectively. Stride, swing and stance time as well as heel and toe contact times are estimated up to ${\pm 0.07}$, ${\pm0.05}$, ${\pm 0.07}$, ${\pm0.07}$ and ${\pm0.12}$ s respectively. This is comparable to and in parts outperforming or defining state-of-the-art. Our results further indicate that the proposed change in methodology could substitute assumption-driven double-integration methods and enable mobile assessment of spatio-temporal stride parameters in clinically critical situations as e.g. in the case of spastic gait impairments.
Parkinson's disease (PD) is a non-curable disease that commonly found among elders that greatly reduce their quality of life. PD primarily affects the gait pattern and slowly changes the walking gait from the normality to disability. The early diagnosing of PD is important for treatments and gait pattern analysis is used as a technique to diagnose PD. The present paper has identified the raw spatiotemporal ground reaction force (GRF) as a key parameter to identify the changes in human gait patterns associated with PD. The changes in GRF are identified using a convolutional neural network throu…
In this paper, a performance evaluation of well-known deep learning models in gait recognition is presented. For this purpose, the transfer learning scheme is adopted to pre-trained models in order to fit the models to the CASIA-B dataset for solving a gait recognition task. In this context, 18 popular Convolutional Neural Networks (CNNs), were re-trained using Gait Energy Images (GEIs) of CASIA-B containing almost 14000 images of 124 classes under various conditions, and their performance was studied in terms of accuracy. Moreover, the performance of the studied models is managed to be explai…
A desireable property of accelerometric gait-based identification systems is robustness to new device orientations presented by users during testing but unseen during the training phase. However, traditional Convolutional neural networks (CNNs) used in these systems compensate poorly for such transformations. In this paper, we target this problem by introducing Quaternion CNN, a network architecture which is intrinsically layer-wise equivariant and globally invariant under 3D rotations of an array of input vectors. We show empirically that this network indeed significantly outperforms a tradit…
Here, we present IDNet, a user authentication framework from smartphone-acquired motion signals. Its goal is to recognize a target user from their way of walking, using the accelerometer and gyroscope (inertial) signals provided by a commercial smartphone worn in the front pocket of the user's trousers. IDNet features several innovations including: i) a robust and smartphone-orientation-independent walking cycle extraction block, ii) a novel feature extractor based on convolutional neural networks, iii) a one-class support vector machine to classify walking cycles, and the coherent integration…
Gait-Based Hand Load Estimation via Deep Latent Variable Models with Auxiliary Information
Department of Systems and Information Engineering, University of Virginia（系统与信息工程系，弗吉尼亚大学）；Department of Industrial and Systems Engineering, Virginia Polytechnic Institute and State University（工业与系统工程系，弗吉尼亚理工学院和州立大学）
EEG2GAIT: A Hierarchical Graph Convolutional Network for EEG-based Gait Decoding
College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）；Institute of Neural Engineering, Graz University of Technology（技术大学神经工程研究所）；Center for Al in Medicine (C-AlM)（医学人工智能中心）
Babes–Bolyai University（巴贝什-博雅依大学）；Sapientia Hungarian University of Transylvania（特兰西瓦尼亚匈牙利语大学）
Recent advances in pattern matching, such as speech or object recognition support the viability of feature learning with deep learning solutions for gait recognition. Past papers have evaluated deep neural networks trained in a supervised manner for this task. In this work, we investigated both supervised and unsupervised approaches. Feature extractors using similar architectures incorporated into end-to-end models and autoencoders were compared based on their ability of learning good representations for a gait verification system. Both feature extractors were trained on the IDNet dataset then…
We consider the problem of identifying people on the basis of their walk (gait) pattern. Classical approaches to tackle this problem are based on, e.g., video recordings or piezoelectric sensors embedded in the floor. In this work, we rely on acoustic and vibration measurements, obtained from a microphone and a geophone sensor, respectively. The contribution of this work is twofold. First, we propose a feature extraction method based on an (untrained) shallow scattering network, specially tailored for the gait signals. Second, we demonstrate that fusing the two modalities improves identificati…
Ulm University（乌尔姆大学）；Ulm University of Applied Sciences（乌尔姆应用科学大学）
To improve the understanding of human gait and to facilitate novel developments in gait rehabilitation, the neural correlates of human gait as measured by means of non-invasive electroencephalography (EEG) have been investigated recently. Particularly, gait-related event-related brain potentials (gERPs) may provide information about the functional role of cortical brain regions in human gait control. The purpose of this paper is to explore possible experimental and technical solutions for time-sensitive analysis of human gait-related ERPs during spontaneous and instructed treadmill walking. A …
University of North Carolina at Charlotte（北卡罗来纳大学夏洛特分校）；Idaho National Laboratory（爱达荷国家实验室）；University at Buffalo（布法罗大学）
Gait is a person's natural walking style and a complex biological process that is unique to each person. Recently, the channel state information (CSI) of WiFi devices have been exploited to capture human gait biometrics for user identification. However, the performance of existing CSI-based gait identification systems is far from satisfactory. They can only achieve limited identification accuracy (maximum $93\%$) only for a very small group of people (i.e., between 2 to 10). To address such challenge, an end-to-end deep CSI learning system is developed, which exploits deep neural networks to a…
Institute of Electronics, National Chiao Tung University（国立交通大学电子研究所）
This paper presents a convolutional neural network based foot motion tracking with only six-axis Inertial-Measurement-Unit (IMU) sensor data. The presented approach can adapt to various walking conditions by adopting differential and window based input. The training data are further augmented by sliding and random window samplings on IMU sensor data to increase data diversity for better performance. The proposed approach fuses predictions of three dimensional output into one model. The proposed fused model can achieve average error of 2.30+-2.23 cm in X-axis, 0.91+-0.95 cm in Y-axis and 0.58+-…
National Chiao Tung University（国立交通大学）；National Taiwan Normal University（国立台湾师范大学）
Previous gait phase detection as convolutional neural network (CNN) based classification task requires cumbersome manual setting of time delay or heavy overlapped sliding windows to accurately classify each phase under different test cases, which is not suitable for streaming Inertial-Measurement-Unit (IMU) sensor data and fails to adapt to different scenarios. This paper presents a segmentation based gait phase detection with only a single six-axis IMU sensor, which can easily adapt to both walking and running at various speeds. The proposed segmentation uses CNN with gait phase aware recepti…
Compared to other biometrics, gait is difficult to conceal and has the advantage of being unobtrusive. Inertial sensors, such as accelerometers and gyroscopes, are often used to capture gait dynamics. These inertial sensors are commonly integrated into smartphones and are widely used by the average person, which makes gait data convenient and inexpensive to collect. In this paper, we study gait recognition using smartphones in the wild. In contrast to traditional methods, which often require a person to walk along a specified road and/or at a normal walking speed, the proposed method collects …
Federal Institute of São Carlos - UFSCar（圣卡洛斯联邦学院）；Eldorado Research Institute（埃尔多拉多研究院）；São Paulo State University - UNESP（圣保罗州立大学）
In general, biometry-based control systems may not rely on individual expected behavior or cooperation to operate appropriately. Instead, such systems should be aware of malicious procedures for unauthorized access attempts. Some works available in the literature suggest addressing the problem through gait recognition approaches. Such methods aim at identifying human beings through intrinsic perceptible features, despite dressed clothes or accessories. Although the issue denotes a relatively long-time challenge, most of the techniques developed to handle the problem present several drawbacks r…
University of Toronto（多伦多大学）；Toronto Rehabilitation Institute, University Health Network（大学健康网络多伦多康复研究所）
This paper presents the design of a wireless sensor network for detecting and alerting the freezing of gait (FoG) symptoms in patients with Parkinson's disease. Three sensor nodes, each integrating a 3-axis accelerometer, can be placed on a patient at ankle, thigh, and truck. Each sensor node can independently detect FoG using an on-device deep learning (DL) model, featuring a squeeze and excitation convolutional neural network (CNN). In a validation using a public dataset, the prototype developed achieved a FoG detection sensitivity of 88.8% and an F1 score of 85.34%, using less than 20 k tra…
Event Extraction plays an important role in information-extraction to understand the world. Event extraction could be split into two subtasks: one is event trigger extraction, the other is event arguments extraction. However, the F-Score of event arguments extraction is much lower than that of event trigger extraction, i.e. in the most recent work, event trigger extraction achieves 80.7%, while event arguments extraction achieves only 58%. In pipelined structures, the difficulty of event arguments extraction lies in its lack of classification feature, and the much higher computation consumptio…
Diagnosing Parkinson's disease is a complex task that requires the evaluation of several motor and non-motor symptoms. During diagnosis, gait abnormalities are among the important symptoms that physicians should consider. However, gait evaluation is challenging and relies on the expertise and subjectivity of clinicians. In this context, the use of an intelligent gait analysis algorithm may assist physicians in order to facilitate the diagnosis process. This paper proposes a novel intelligent Parkinson detection system based on deep learning techniques to analyze gait information. We used 1D co…
Haute-Ecole Arc Santé, HES-SO University of Applied Sciences and Arts Western Switzerland（瑞士西部应用科学与艺术大学阿尔克卫生学院）；University Hospitals of Geneva（日内瓦大学医院）
The fact that every human has a distinctive walking style has prompted a proposal to use gait recognition as an identification criterion. Using end-to-end learning, I investigated whether the center-of-pressure trajectory is sufficiently unique to identify a person with a high certainty. Thirty-six adults walked on a treadmill equipped with a force platform that recorded the positions of the center of pressure. The raw two-dimensional signals were sliced into segments of two gait cycles. A set of 20,250 segments from 30 subjects was used to configure and train convolutional neural networks (CN…
3D Gait-Based Autism Classification Using Attention-Enhanced Deep Learning with Cross-Fold Statistical Stability Analysis
Begum Rokeya University（贝古姆罗基亚大学）；Texas State University（德克萨斯州立大学）；Asian Institute of Technology（亚洲理工学院）；National Center for Geriatrics and Gerontology（国立长寿医疗研究中心）
Singapore University of Technology and Design（新加坡科技设计大学）
Dependency trees convey rich structural information that is proven useful for extracting relations among entities in text. However, how to effectively make use of relevant information while ignoring irrelevant information from the dependency trees remains a challenging research question. Existing approaches employing rule based hard-pruning strategies for selecting relevant partial dependency structures may not always yield optimal results. In this work, we propose Attention Guided Graph Convolutional Networks (AGGCNs), a novel model which directly takes full dependency trees as inputs. Our mo…
We report an implementation of a clinical information extraction tool that leverages deep neural network to annotate event spans and their attributes from raw clinical notes and pathology reports. Our approach uses context words and their part-of-speech tags and shape information as features. Then we hire temporal (1D) convolutional neural network to learn hidden feature representations. Finally, we use Multilayer Perceptron (MLP) to predict event spans. The empirical evaluation demonstrates that our approach significantly outperforms baselines.
Technische Universität Dresden（德累斯顿工业大学）；King Abdullah University of Science and Technology (KAUST)（阿卜杜拉国王科技大学）；University of Padova（帕多瓦大学）；M31
Smartphones and wearable devices are fast growing technologies that, in conjunction with advances in wireless sensor hardware, are enabling ubiquitous sensing applications. Wearables are suitable for indoor and outdoor scenarios, can be placed on many parts of the human body and can integrate a large number of sensors capable of gathering physiological and behavioral biometric information. Here, we are concerned with gait analysis systems that extract meaningful information from a user's movements to identify anomalies and changes in their walking style. The solution that is put forward is sub…
Deep Convolutional Neural Networks with Zero-Padding: Feature Extraction and Learning
Shenyang Institute of Automation, Chinese Academy of Sciences（中国科学院沈阳自动化研究所）；Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences（中国科学院机器人与智能制造创新研究院）；Xi’an Jiaotong University（西安交通大学）；University of Sydney（悉尼大学）
TU Kaiserslautern（凯泽斯劳滕工业大学）；University of Mainz（美因茨大学）；Johannes Gutenberg University of Mainz（约翰内斯·古腾堡大学美因茨分校）；TU Darmstadt（达姆施塔特工业大学）
Most deep neural networks are considered to be black boxes, meaning their output is hard to interpret. In contrast, logical expressions are considered to be more comprehensible since they use symbols that are semantically close to natural language instead of distributed representations. However, for high-dimensional input data such as images, the individual symbols, i.e. pixels, are not easily interpretable. We introduce the concept of first-order convolutional rules, which are logical rules that can be extracted using a convolutional neural network (CNN), and whose complexity depends on the s…
Amirkabir University of Technology（阿米卡比尔理工大学）；Chalmers University of Technology（查尔姆斯理工大学）
Spiking neural networks (SNNs) can be used in low-power and embedded systems (such as emerging neuromorphic chips) due to their event-based nature. Also, they have the advantage of low computation cost in contrast to conventional artificial neural networks (ANNs), while preserving ANN's properties. However, temporal coding in layers of convolutional spiking neural networks and other types of SNNs has yet to be studied. In this paper, we provide insight into spatio-temporal feature extraction of convolutional SNNs in experiments designed to exploit this property. The shallow convolutional SNN o…
Knowledge extraction is used to convert neural networks into symbolic descriptions with the objective of producing more comprehensible learning models. The central challenge is to find an explanation which is more comprehensible than the original model while still representing that model faithfully. The distributed nature of deep networks has led many to believe that the hidden features of a neural network cannot be explained by logical descriptions simple enough to be comprehensible. In this paper, we propose a novel layerwise knowledge extraction method using M-of-N rules which seeks to obta…
Stock prediction is a topic undergoing intense study for many years. Finance experts and mathematicians have been working on a way to predict the future stock price so as to decide to buy the stock or sell it to make profit. Stock experts or economists, usually analyze on the previous stock values using technical indicators, sentiment analysis etc to predict the future stock price. In recent years, many researches have extensively used machine learning for predicting the stock behaviour. In this paper we propose data driven deep learning approach to predict the future stock value with the prev…
College of Computer and Data Science, Fuzhou University（福州大学计算机与数据科学学院）
In recent years, some traffic information prediction methods have been proposed to provide the precise information of travel time, vehicle speed, and traffic flow for highways. However, big errors may be obtained by these methods for urban roads or the alternative roads of highways. Therefore, this study proposes a travel time prediction method based on convolutional neural networks to extract important factors for the improvement of traffic information prediction. In practical experimental environments, the travel time records of No. 5 Highway and the alternative roads of its were collected a…
Accurate capacitance extraction is becoming more important for designing integrated circuits under advanced process technology. The pattern matching based full-chip extraction methodology delivers fast computational speed, but suffers from large error, and tedious efforts on building capacitance models of the increasing structure patterns. In this work, we propose an effective method for building convolutional neural network (CNN) based capacitance models (called CNN-Cap) for two-dimensional (2-D) structures in full-chip capacitance extraction. With a novel grid-based data representation, the …
Centre Borelli（博雷利中心）；CNRS（法国国家科学研究中心）；ENS Paris Saclay（巴黎-萨克雷高等师范学院）；Université de Paris（巴黎大学）；SSA（法国武装部队卫生服务局）；Hôpital Necker-Enfants Malades（内克尔-患病儿童医院）；AP-HP（巴黎公立医院集团）；Université Sorbonne Paris Nord（索邦巴黎北大学）；University Paris-Saclay（巴黎-萨克雷大学）；Inria（法国国家信息与自动化研究所）；CEA（法国原子能和替代能源委员会）
The analysis of the Nystagmus waveforms from eye-tracking records is crucial for the clinicial interpretation of this pathological movement. A major issue to automatize this analysis is the presence of natural eye movements and eye blink artefacts that are mixed with the signal of interest. We propose a method based on Convolutional Dictionary Learning that is able to automaticcaly highlight the Nystagmus waveforms, separating the natural motion from the pathological movements. We show on simulated signals that our method can indeed improve the pattern recovery rate and provide clinical exampl…
Deep neural networks are a powerful tool for feature learning and extraction given their ability to model high-level abstractions in highly complex data. One area worth exploring in feature learning and extraction using deep neural networks is efficient neural connectivity formation for faster feature learning and extraction. Motivated by findings of stochastic synaptic connectivity formation in the brain as well as the brain's uncanny ability to efficiently represent information, we propose the efficient learning and extraction of features via StochasticNets, where sparsely-connected deep neu…
The last decade has witnessed the success of the traditional feature-based method on exploiting the discrete structures such as words or lexical patterns to extract relations from text. Recently, convolutional and recurrent neural networks has provided very effective mechanisms to capture the hidden structures within sentences via continuous representations, thereby significantly advancing the performance of relation extraction. The advantage of convolutional neural networks is their capacity to generalize the consecutive k-grams in the sentences while recurrent neural networks are effective t…
Improvement of Performance in Freezing of Gait detection in Parkinsons Disease using Transformer networks and a single waist worn triaxial accelerometer
Universidad Politécnica de Madrid（马德里理工大学）；University of Minho（米尼奥大学）；Politecnico di Torino（都灵理工大学）
Clova AI Research（Clova人工智能研究院）；NAVER Corp.（NAVER公司）；Hong Kong University of Science and Technology（香港科技大学）
Music highlights are valuable contents for music services. Most methods focused on low-level signal features. We propose a method for extracting highlights using high-level features from convolutional recurrent attention networks (CRAN). CRAN utilizes convolution and recurrent layers for sequential learning with an attention mechanism. The attention allows CRAN to capture significant snippets for distinguishing between genres, thus being used as a high-level feature. CRAN was evaluated on over 32,000 popular tracks in Korea for two months. Experimental results show our method outperforms three…
Recent work for extracting relations from texts has achieved excellent performance. However, most existing methods pay less attention to the efficiency, making it still challenging to quickly extract relations from massive or streaming text data in realistic scenarios. The main efficiency bottleneck is that these methods use a Transformer-based pre-trained language model for encoding, which heavily affects the training speed and inference speed. To address this issue, we propose a fast relation extraction model (FastRE) based on convolutional encoder and improved cascade binary tagging framewo…
Research on Deep Learning Model of Feature Extraction Based on Convolutional Neural Network
Washington University（华盛顿大学）；Carnegie Mellon University（卡内基梅隆大学）；New York University（纽约大学）；University of Houston（休斯顿大学）；University of Southern California（南加州大学）
University of Tehran（德黑兰大学）；Polytechnic Institute of Tomar（托马尔理工学院）；Kyoto University（京都大学）；INESC Technology and Science（INESC技术与科学研究院）
Event extraction (EE) is one of the core information extraction tasks, whose purpose is to automatically identify and extract information about incidents and their actors from texts. This may be beneficial to several domains such as knowledge bases, question answering, information retrieval and summarization tasks, to name a few. The problem of extracting event information from texts is longstanding and usually relies on elaborately designed lexical and syntactic features, which, however, take a large amount of human effort and lack generalization. More recently, deep neural network approaches…
Deep convolutional neural networks have led to breakthrough results in numerous practical machine learning tasks such as classification of images in the ImageNet data set, control-policy-learning to play Atari games or the board game Go, and image captioning. Many of these applications first perform feature extraction and then feed the results thereof into a trainable classifier. The mathematical analysis of deep convolutional neural networks for feature extraction was initiated by Mallat, 2012. Specifically, Mallat considered so-called scattering networks based on a wavelet transform followed…
Georgia Institute of Technology（佐治亚理工学院）；Brookhaven National Laboratory（布鲁克海文国家实验室）；IBM Research（IBM研究院）；University of Enna（恩纳大学）；NTNU（挪威科技大学）
We propose a novel decentralized feature extraction approach in federated learning to address privacy-preservation issues for speech recognition. It is built upon a quantum convolutional neural network (QCNN) composed of a quantum circuit encoder for feature extraction, and a recurrent neural network (RNN) based end-to-end acoustic model (AM). To enhance model parameter protection in a decentralized architecture, an input speech is first up-streamed to a quantum computing server to extract Mel-spectrogram, and the corresponding convolutional features are encoded using a quantum circuit algorit…
