A Deep Learning Method for Complex Human Activity Recognition Using Virtual Wearable Sensors
Shanghai Jiao Tong University（上海交通大学）；School of Electrical Information and Electrical Engineering（电子信息与电气工程学院）；Shanghai Key Laboratory of Navigation and Location-based Services（上海市导航与位置服务重点实验室）
Fanyi Xiao, Ling Pei, Lei Chu, Danping Zou, Wenxian Yu, Yifan Zhu, Tao Li
Sensor-based human activity recognition (HAR) is now a research hotspot in multiple application areas. With the rise of smart wearable devices equipped with inertial measurement units (IMUs), researchers begin to utilize IMU data for HAR. By employing machine learning algorithms, early IMU-based research for HAR can achieve accurate classification results on traditional classical HAR datasets, containing only simple and repetitive daily activities. However, these datasets rarely display a rich diversity of information in real-scene. In this paper, we propose a novel method based on deep learning for complex HAR in the real-scene. Specially, in the off-line training stage, the AMASS dataset, containing abundant human poses and virtual IMU data, is innovatively adopted for enhancing the variety and diversity. Moreover, a deep convolutional neural network with an unsupervised penalty is proposed to automatically extract the features of AMASS and improve the robustness. In the on-line testing stage, by leveraging advantages of the transfer learning, we obtain the final result by fine-tuning the partial neural network (optimizing the parameters in the fully-connected layers) using the real IMU data. The experimental results show that the proposed method can surprisingly converge in a few iterations and achieve an accuracy of 91.15% on a real IMU dataset, demonstrating the efficiency and effectiveness of the proposed method.
Together with the rapid development of the Internet of Things (IoT), human activity recognition (HAR) using wearable Inertial Measurement Units (IMUs) becomes a promising technology for many research areas. Recently, deep learning-based methods pave a new way of understanding and performing analysis of the complex data in the HAR system. However, the performance of these methods is mostly based on the quality and quantity of the collected data. In this paper, we innovatively propose to build a large database based on virtual IMUs and then address technical issues by introducing a multiple-doma…
German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）；Technische Universität Kaiserslautern（凯泽斯劳滕工业大学）
Wearable sensor-based human activity recognition (HAR) has emerged as a principal research area and is utilized in a variety of applications. Recently, deep learning-based methods have achieved significant improvement in the HAR field with the development of human-computer interaction applications. However, they are limited to operating in a local neighborhood in the process of a standard convolution neural network, and correlations between different sensors on body positions are ignored. In addition, they still face significant challenging problems with performance degradation due to large ga…
VTT Technical Research Centre of Finland（芬兰VTT技术研究中心）；University of Oulu（奥卢大学）
Human Activity Recognition (HAR) from wearable sensor data identifies movements or activities in unconstrained environments. HAR is a challenging problem as it presents great variability across subjects. Obtaining large amounts of labelled data is not straightforward, since wearable sensor signals are not easy to label upon simple human inspection. In our work, we propose the use of neural networks for the generation of realistic signals and features using human activity monocular videos. We show how these generated features and signals can be utilized, instead of their real counterparts, to t…
The University of Adelaide（阿德莱德大学）；China University of Mining and Technology（中国矿业大学）
Recognizing human activities in a sequence is a challenging area of research in ubiquitous computing. Most approaches use a fixed size sliding window over consecutive samples to extract features---either handcrafted or learned features---and predict a single label for all samples in the window. Two key problems emanate from this approach: i) the samples in one window may not always share the same label. Consequently, using one label for all samples within a window inevitably lead to loss of information; ii) the testing phase is constrained by the window size selected during training while the …
SWL-Adapt: An Unsupervised Domain Adaptation Model with Sample Weight Learning for Cross-User Wearable Human Activity Recognition
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Alibaba-Zhejiang University Joint Research Institute of Frontier Technologies（阿里巴巴-浙江大学前沿技术联合研究院）
Toronto Rehabilitation Institute（多伦多康复研究所）；University of Toronto（多伦多大学）；uhn（多伦多大学健康网络）
A fall is an abnormal activity that occurs rarely, so it is hard to collect real data for falls. It is, therefore, difficult to use supervised learning methods to automatically detect falls. Another challenge in using machine learning methods to automatically detect falls is the choice of engineered features. In this paper, we propose to use an ensemble of autoencoders to extract features from different channels of wearable sensor data trained only on normal activities. We show that the traditional approach of choosing a threshold as the maximum of the reconstruction error on the training norm…
Human activity recognition based on wearable sensor data has been an attractive research topic due to its application in areas such as healthcare and smart environments. In this context, many works have presented remarkable results using accelerometer, gyroscope and magnetometer data to represent the activities categories. However, current studies do not consider important issues that lead to skewed results, making it hard to assess the quality of sensor-based human activity recognition and preventing a direct comparison of previous works. These issues include the samples generation processes …
Nanjing Normal University（南京师范大学）；Yunnan University（云南大学）；Nanjing University of Information Science and Technology（南京信息工程大学）
Recently, deep learning has represented an important research trend in human activity recognition (HAR). In particular, deep convolutional neural networks (CNNs) have achieved state-of-the-art performance on various HAR datasets. For deep learning, improvements in performance have to heavily rely on increasing model size or capacity to scale to larger and larger datasets, which inevitably leads to the increase of operations. A high number of operations in deep leaning increases computational cost and is not suitable for real-time HAR using mobile and wearable sensors. Though shallow learning t…
Arizona State University（亚利桑那州立大学）；Lonnie and Muhammad Ali Movement Disorder Center（朗尼与穆罕默德·阿里运动障碍中心）
Human activity recognition~(HAR) has attracted significant research interest due to its applications in health monitoring and patient rehabilitation. Recent research on HAR focuses on using smartphones due to their widespread use. However, this leads to inconvenient use, limited choice of sensors and inefficient use of resources, since smartphones are not designed for HAR. This paper presents the first HAR framework that can perform both online training and inference. The proposed framework starts with a novel technique that generates features using the fast Fourier and discrete wavelet transf…
Wearable devices are increasingly used, thanks to the wide set of applications that can be deployed exploiting their ability to monitor physical activity and health-related parameters. Their usage has been recently proposed to perform biometric recognition, leveraging on the uniqueness of the recorded traits to generate discriminative identifiers. Most of the studies conducted on this topic have considered signals derived from cardiac activity, detecting it mainly using electrical measurements thorugh electrocardiography, or optical recordings employing photoplethysmography. In this paper we i…
Wearable-based behaviour interpolation for semi-supervised human activity recognition
Durham University（杜伦大学）；Newcastle University（纽卡斯尔大学）；Tencent YouTu Lab（腾讯优图实验室）；Chinese Academy of Sciences R&D Center for Internet of Things（中国科学院物联网研究发展中心）
Shanghai Jiao Tong University（上海交通大学）；Huazhong University of Science and Technology（华中科技大学）
Human activity recognition (HAR) with wearables is promising research that can be widely adopted in many smart healthcare applications. In recent years, the deep learning-based HAR models have achieved impressive recognition performance. However, most HAR algorithms are susceptible to the multi-class windows problem that is essential yet rarely exploited. In this paper, we propose to relieve this challenging problem by introducing the segmentation technology into HAR, yielding joint activity segmentation and recognition. Especially, we introduce the Multi-Stage Temporal Convolutional Network (…
MoPFormer: Motion-Primitive Transformer for Wearable-Sensor Activity Recognition
Southern University of Science and Technology（南方科技大学）；City University of Hong Kong（香港城市大学）；Zhejiang University（浙江大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）
Bangladesh University of Engineering and Technology（孟加拉国工程技术大学）
Activity detection from first-person videos (FPV) captured using a wearable camera is an active research field with potential applications in many sectors, including healthcare, law enforcement, and rehabilitation. State-of-the-art methods use optical flow-based hybrid techniques that rely on features derived from the motion of objects from consecutive frames. In this work, we developed a two-stream network, the \emph{SegCodeNet}, that uses a network branch containing video-streams with color-coded semantic segmentation masks of relevant objects in addition to the original RGB video-stream. We…
To help the blind people walk to the destination efficiently and safely in indoor environment, a novel wearable navigation device is presented in this paper. The locating, way-finding, route following and obstacle avoiding modules are the essential components in a navigation system, while it remains a challenging task to consider obstacle avoiding during route following, as the indoor environment is complex, changeable and possibly with dynamic objects. To address this issue, we propose a novel scheme which utilizes a dynamic sub-goal selecting strategy to guide the users to the destination an…
Newcastle University（纽卡斯尔大学）；Georgia Institute of Technology（佐治亚理工学院）
Recently, deep learning (DL) methods have been introduced very successfully into human activity recognition (HAR) scenarios in ubiquitous and wearable computing. Especially the prospect of overcoming the need for manual feature design combined with superior classification capabilities render deep neural networks very attractive for real-life HAR application. Even though DL-based approaches now outperform the state-of-the-art in a number of recognitions tasks of the field, yet substantial challenges remain. Most prominently, issues with real-life datasets, typically including imbalanced dataset…
Nanjing Normal University（南京师范大学）；Nanjing University of Information Science and Technology（南京信息工程大学）
Recently, convolutional neural networks (CNNs) have set latest state-of-the-art on various human activity recognition (HAR) datasets. However, deep CNNs often require more computing resources, which limits their applications in embedded HAR. Although many successful methods have been proposed to reduce memory and FLOPs of CNNs, they often involve special network architectures designed for visual tasks, which are not suitable for deep HAR tasks with time series sensor signals, due to remarkable discrepancy. Therefore, it is necessary to develop lightweight deep models to perform HAR. As filter …
Nanjing Normal University（南京师范大学）；Nanjing University of Information Science and Technology（南京信息工程大学）；Yunnan University（云南大学）
Human activity recognition (HAR) in ubiquitous computing has been beginning to incorporate attention into the context of deep neural networks (DNNs), in which the rich sensing data from multimodal sensors such as accelerometer and gyroscope is used to infer human activities. Recently, two attention methods are proposed via combining with Gated Recurrent Units (GRU) and Long Short-Term Memory (LSTM) network, which can capture the dependencies of sensing signals in both spatial and temporal domains simultaneously. However, recurrent networks often have a weak feature representing power compared …
A comparative study on wearables and single-camera video for upper-limb out-of-thelab activity recognition with different deep learning architectures
University of Valladolid（巴利亚多利德大学）；University of Geneva（日内瓦大学）；University of Applied Sciences Western Switzerland (HES-SO) Valais-Wallis（瑞士西部应用科学与艺术大学（HES-SO）瓦莱-沃利斯）
Air Force Engineering University（空军工程大学）；Laval University（拉瓦尔大学）；Xidian University（西安电子科技大学）
Human activity recognition (HAR) has become a popular topic in research because of its wide application. With the development of deep learning, new ideas have appeared to address HAR problems. Here, a deep network architecture using residual bidirectional long short-term memory (LSTM) cells is proposed. The advantages of the new network include that a bidirectional connection can concatenate the positive time direction (forward state) and the negative time direction (backward state). Second, residual connections between stacked cells act as highways for gradients, which can pass underlying inf…
P2LHAP:Wearable sensor-based human activity recognition, segmentation and forecast through Patch-to-Label Seq2Seq Transformer
School of Computer Science, University of South China（南华大学计算机学院）；School of Computer Science and Technology, Dalian University of Technology（大连理工大学计算机科学与技术学院）；Department of Computer & Communication Engineering, University of Science and Technology Beijing（北京科技大学计算机与通信工程学院）
HARMamba: Efficient and Lightweight Wearable Sensor Human Activity Recognition Based on Bidirectional Mamba
School of Computer Science, University of South China（南华大学计算机学院）；School of Computer Science and Technology, Dalian University of Technology（大连理工大学计算机科学与技术学院）；Department of Computer & Communication Engineering, University of Science and Technology Beijing（北京科技大学计算机与通信工程学院）；School of Computing, Ulster University（阿尔斯特大学计算学院）
Decomposing and Fusing Intra- and Inter-Sensor Spatio-Temporal Signal for Multi-Sensor Wearable Human Activity Recognition
School of Big Data and Software Engineering, Chongqing University（重庆大学大数据与软件学院）；Center for Data Science, Peking University（北京大学数据科学中心）；School of Mathematical Sciences, Peking University（北京大学数学科学学院）；School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）
