We propose cross-modal attentive connections, a new dynamic and effective technique for multimodal representation learning from wearable data. Our solution can be integrated into any stage of the pipeline, i.e., after any convolutional layer or block, to create intermediate connections between individual streams responsible for processing each modality. Additionally, our method benefits from two properties. First, it can share information uni-directionally (from one modality to the other) or bi-directionally. Second, it can be integrated into multiple stages at the same time to further allow network gradients to be exchanged in several touch-points. We perform extensive experiments on three public multimodal wearable datasets, WESAD, SWELL-KW, and CASE, and demonstrate that our method can effectively regulate and share information between different modalities to learn better representations. Our experiments further demonstrate that once integrated into simple CNN-based multimodal solutions (2, 3, or 4 modalities), our method can result in superior or competitive performance to state-of-the-art and outperform a variety of baseline uni-modal and classical multimodal methods.
School of Biomedical Engineering, Medical School, Shenzhen University（深圳大学医学院生物医学工程学院）；Guangdong Provincial Key Laboratory of Biomedical Measurements and Ultrasound Imaging（广东省生物医学测量与超声成像重点实验室）；Institute of Computing and Intelligence, Harbin Institute of Technology（哈尔滨工业大学计算与智能研究院）；Peng Cheng Laboratory（鹏城实验室）；Shenzhen Mental Health Center, Shenzhen Kangning Hospital（深圳市精神卫生中心（深圳市康宁医院））
Queen’s University（女王大学）；Ingenuity Labs Research Institute（创新实验室研究所）
Classification of human emotions can play an essential role in the design and improvement of human-machine systems. While individual biological signals such as Electrocardiogram (ECG) and Electrodermal Activity (EDA) have been widely used for emotion recognition with machine learning methods, multimodal approaches generally fuse extracted features or final classification/regression results to boost performance. To enhance multimodal learning, we present a novel attentive cross-modal connection to share information between convolutional neural networks responsible for learning individual modali…
With recent developments in smart technologies, there has been a growing focus on the use of artificial intelligence and machine learning for affective computing to further enhance the user experience through emotion recognition. Typically, machine learning models used for affective computing are trained using manually extracted features from biological signals. Such features may not generalize well for large datasets and may be sub-optimal in capturing the information from the raw input data. One approach to address this issue is to use fully supervised deep learning methods to learn latent r…
Speech Emotion Recognition (SER) task has known significant improvements over the last years with the advent of Deep Neural Networks (DNNs). However, even the most successful methods are still rather failing when adaptation to specific speakers and scenarios is needed, inevitably leading to poorer performances when compared to humans. In this paper, we present novel work based on the idea of teaching the emotion recognition network about speaker identity. Our system is a combination of two ACRNN classifiers respectively dedicated to speaker and emotion recognition. The first informs the latter…
Wrocław University of Science and Technology（弗罗茨瓦夫理工大学）；Capgemini（凯捷）
Wearables like smartwatches or wrist bands equipped with pervasive sensors enable us to monitor our physiological signals. In this study, we address the question whether they can help us to recognize our emotions in our everyday life for ubiquitous computing. Using the systematic literature review, we identified crucial research steps and discussed the main limitations and problems in the domain.
Spectrogram is commonly used as the input feature of deep neural networks to learn the high(er)-level time-frequency pattern of speech signal for speech emotion recognition (SER). \textcolor{black}{Generally, different emotions correspond to specific energy activations both within frequency bands and time frames on spectrogram, which indicates the frequency and time domains are both essential to represent the emotion for SER. However, recent spectrogram-based works mainly focus on modeling the long-term dependency in time domain, leading to these methods encountering the following two issues: …
Seoul National University（首尔大学）；Idiap Research Institute（伊迪亚普研究所）
In this work, we explore the impact of visual modality in addition to speech and text for improving the accuracy of the emotion detection system. The traditional approaches tackle this task by fusing the knowledge from the various modalities independently for performing emotion classification. In contrast to these approaches, we tackle the problem by introducing an attention mechanism to combine the information. In this regard, we first apply a neural network to obtain hidden representations of the modalities. Then, the attention mechanism is defined to select and aggregate important parts of …
Wearable data from subjects playing Super Mario, sitting university exams, or performing physical exercise help detect acute mood episodes via self-supervised learning
School of Informatics, University of Edinburgh（爱丁堡大学信息学院）；Hospital Clínic de Barcelona, University of Barcelona（巴塞罗那大学巴塞罗那临床医院）；Institute of Psychiatry, Psychology and Neuroscience, King’s College London（伦敦国王学院精神病学、心理学与神经科学研究所）；Division of Psychiatry, University of Edinburgh（爱丁堡大学精神病学系）
Cyprus University of Technology（塞浦路斯理工大学）；Telefonica Research（西班牙电信研究院）；Samsung Research（三星研究院）
Modern deep learning approaches have achieved groundbreaking performance in modeling and classifying sequential data. Specifically, attention networks constitute the state-of-the-art paradigm for capturing long temporal dynamics. This paper examines the efficacy of this paradigm in the challenging task of emotion recognition in dyadic conversations. In contrast to existing approaches, our work introduces a novel attention mechanism capable of inferring the immensity of the effect of each past utterance on the current speaker emotional state. The proposed attention mechanism performs this infer…
Emotion Recognition in Older Adults with Quantum Machine Learning and Wearable Sensors
Department of Electrical Engineering and Computer Science（电气工程与计算机科学系）；University of Tennessee, Knoxville, TN, USA（田纳西大学，诺克斯维尔分校）；Quantum Science Center, Oak Ridge National Laboratory, Oak Ridge, TN, USA（橡树岭国家实验室量子科学中心）
Universitat Politècnica de Catalunya（加泰罗尼亚理工大学）；Eurecat, Centre Tecnològic de Catalunya（加泰罗尼亚技术中心（Eurecat））；Universitat Oberta de Catalunya（加泰罗尼亚开放大学）
We present MDEAW, a multimodal database consisting of Electrodermal Activity (EDA) and Photoplethysmography (PPG) signals recorded during the exams for the course taught by the teacher at Eurecat Academy, Sabadell, Barcelona in order to elicit the emotional reactions to the students in a classroom scenario. Signals from 10 students were recorded along with the students' self-assessment of their affective state after each stimulus, in terms of 6 basic emotion states. All the signals were captured using portable, wearable, wireless, low-cost, and off-the-shelf equipment that has the potential to…
RMIT University（皇家墨尔本理工大学）；Swinburne University of Technology（斯威本科技大学）；University of New South Wales (UNSW)（新南威尔士大学（UNSW））
We conducted a field study at a K-12 private school in the suburbs of Melbourne, Australia. The data capture contained two elements: First, a 5-month longitudinal field study In-Gauge using two outdoor weather stations, as well as indoor weather stations in 17 classrooms and temperature sensors on the vents of occupant-controlled room air-conditioners; these were collated into individual datasets for each classroom at a 5-minute logging frequency, including additional data on occupant presence. The dataset was used to derive predictive models of how occupants operate room air-conditioning unit…
Recording Brain Activity While Listening to Music Using Wearable EEG Devices Combined with Bidirectional Long Short-Term Memory Networks
School of Music, Jiangxi Normal University（江西师范大学音乐学院）；School of Electronic Information, HuZhou College（湖州学院电子信息学院）；San Francisco State University（旧金山州立大学）
Optimizing Emotion Recognition with Wearable Sensor Data: Unveiling Patterns in Body Movements and Heart Rate through Random Forest Hyperparameter Tuning
Queen’s University（女王大学）；Ingenuity Labs Research Institute（英格尼蒂实验室研究所）
Deep learning techniques have considerably improved speech processing in recent years. Speaker representations extracted by deep learning models are being used in a wide range of tasks such as speaker recognition and speech emotion recognition. Attention mechanisms have started to play an important role in improving deep learning models in the field of speech processing. Nonetheless, despite the fact that important speaker-related information can be embedded in individual frequency-bins of the input spectral representations, current attention models are unable to attend to fine-grained informa…
SRM Institute of Science and Technology（SRM科学技术学院）；Samsung Research Institute（三星研究院）
Smart wearables have played an integral part in our day to day life. From recording ECG signals to analysing body fat composition, the smart wearables can do it all. The smart devices encompass various sensors which can be employed to derive meaningful information regarding the user's physical and psychological conditions. Our approach focuses on employing such sensors to identify and obtain the variations in the mood of a user at a given instance through the use of supervised machine learning techniques. The study examines the performance of various supervised learning models such as Decision…
With most recent estimates giving an incidence rate of 1 in 68 children in the United States, the autism spectrum disorder (ASD) is a growing public health crisis. Many of these children struggle to make eye contact, recognize facial expressions, and engage in social interactions. Today the standard for treatment of the core autism-related deficits focuses on a form of behavior training known as Applied Behavioral Analysis. To address perceived deficits in expression recognition, ABA approaches routinely involve the use of prompts such as flash cards for repetitive emotion recognition training…
MASA-TCN: Multi-anchor Space-aware Temporal Convolutional Neural Networks for Continuous and Discrete EEG Emotion Recognition
School of Computer Science and Engineering, Nanyang Technological University（南洋理工大学计算机科学与工程学院）；School of Biological Science and Medical Engineering, Southeast University（东南大学生物科学与医学工程学院）
Northeastern University（东北大学）；University of Pittsburgh（匹兹堡大学）
There is an increasing consensus among re- searchers that making a computer emotionally intelligent with the ability to decode human affective states would allow a more meaningful and natural way of human-computer interactions (HCIs). One unobtrusive and non-invasive way of recognizing human affective states entails the exploration of how physiological signals vary under different emotional experiences. In particular, this paper explores the correlation between autonomically-mediated changes in multimodal body signals and discrete emotional states. In order to fully exploit the information in …
REMONI: An Autonomous System Integrating Wearables and Multimodal Large Language Models for Enhanced Remote Health Monitoring
2 Department of Electrical；Computer Engineering University of Waterloo, Waterloo, ON, Canada N2L 3G1 Email；3 College of Computer；Information Sciences Prince Sultan University Email
University of California Irvine（加州大学尔湾分校）；University of California, Irvine（加州大学尔湾分校）
Monitoring and understanding affective states are important aspects of healthy functioning and treatment of mood-based disorders. Recent advancements of ubiquitous wearable technologies have increased the reliability of such tools in detecting and accurately estimating mental states (e.g., mood, stress, etc.), offering comprehensive and continuous monitoring of individuals over time. Previous attempts to model an individual's mental state were limited to subjective approaches or the inclusion of only a few modalities (i.e., phone, watch). Thus, the goal of our study was to investigate the capa…
Introducing WARM-VR: Benchmark Dataset for Multimodal Wearable Affect Recognition in Virtual Reality
School of Electrical Engineering and Computer Science, University of Ottawa（Ottawa 大学电气工程与计算机科学学院）；Computer Science, University of Ottawa（Ottawa 大学计算机科学系）；Computer Vision Department, Mohamed bin Zayed University of Artificial Intelligence（摩根·bin·扎耶德人工智能大学计算机视觉部门）
