We study multimodal affect modeling when EEG and peripheral physiology are asynchronous, which most fusion methods ignore or handle with costly warping. We propose Cross-Temporal Attention Fusion (CTAF), a self-supervised module that learns soft bidirectional alignments between modalities and builds a robust clip embedding using time-aware cross attention, a lightweight fusion gate, and alignment-regularized contrastive objectives with optional weak supervision. On the K-EmoCon dataset, under leave-one-out cross-validation evaluation, CTAF yields higher cosine margins for matched pairs and better cross-modal token retrieval within one second, and it is competitive with the baseline on three-bin accuracy and macro-F1 while using few labels. Our contributions are a time-aware fusion mechanism that directly models correspondence, an alignment-driven self-supervised objective tailored to EEG and physiology, and an evaluation protocol that measures alignment quality itself. Our approach accounts for the coupling between the central and autonomic nervous systems in psychophysiological time series. These results indicate that CTAF is a strong step toward label-efficient, generalizable EEG-peripheral fusion under temporal asynchrony.
Deception Detection from Linguistic and Physiological Data Streams Using Bimodal Convolutional Neural Networks
University of Michigan（密歇根大学）；Columbia University（哥伦比亚大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Saarland University of Applied Science（萨尔应用科学大学）
University of Cambridge（剑桥大学）；Alan Turing Institute（艾伦·图灵研究所）
To date, research on sensor-equipped mobile devices has primarily focused on the purely supervised task of human activity recognition (walking, running, etc), demonstrating limited success in inferring high-level health outcomes from low-level signals, such as acceleration. Here, we present a novel self-supervised representation learning method using activity and heart rate (HR) signals without semantic labels. With a deep neural network, we set HR responses as the supervisory signal for the activity data, leveraging their underlying physiological relationship. We evaluate our model in the lar…
Omni-Sleep: A Sleep Foundation Model via Hierarchical Contrastive Learning of CNS-ANS Dynamics
Department of Biomedical Engineering, Southern University of Science and Technology（南方科技大学生物医学工程系）；Omni-Intelligence（全知智能）；Shenzhen Loop Area Institute（深圳河套学院）
Xidian University（西安电子科技大学）；Robarts Research Institute（罗伯茨研究所）；Western University（韦仕敦大学）；University of Michigan（密歇根大学）
Electronic health records (EHRs) are usually highly dimensional, heterogeneous, and multimodal. Besides, the random recording of clinical variables results in high missing rates and uneven time intervals between adjacent records in the multivariate clinical time-series data extracted from EHRs. Current works using clinical time-series data for patient representation regard the patients' physiological status as a discrete process described by sporadically collected records. However, changes in the patient's physiological condition are continuous and dynamic processes. The perception of time and…
Transformer-based Self-supervised Multimodal Representation Learning for Wearable Emotion Recognition
Univ. Lille（里尔大学）；CNRS（法国国家科学研究中心）；Centrale Lille（里尔中央理工学院）；Institut Mines-Télécom（法国国立高等矿业电信学校联盟）；IMT Nord Europe（法国国立高等矿业电信学校联盟北欧分校）；INSERM（法国国家健康与医学研究院）；University of Lille（里尔大学）
Characterizing the dynamic interactive patterns of complex systems helps gain in-depth understanding of how components interrelate with each other while performing certain functions as a whole. In this study, we present a novel multimodal data fusion approach to construct a complex network, which models the interactions of biological subsystems in the human body under emotional states through physiological responses. Joint recurrence plot and temporal network metrics are employed to integrate the multimodal information at the signal level. A benchmark public dataset of is used for evaluating o…
Multimodal Machine Learning in Mental Health: A Survey of Data, Algorithms, and Challenges
Queen Mary University of London United Kingdom；Queen Mary University of London \& Jo z ef Stefan Institute United Kingdom \& Slovenia；Queen Mary University of London；Queen Mary University of London \& Jo z ef Stefan Institute
CNRS（法国国家科学研究中心）；University of Lille（里尔大学）；Centrale Lille（里尔中央理工学院）；IMT Nord Europe（法国北部高等矿业电信学院）；Institut Mines-Télécom（法国矿业电信学院）；INSERM（法国国家健康与医学研究院）
Existing multimodal stress/pain recognition approaches generally extract features from different modalities independently and thus ignore cross-modality correlations. This paper proposes a novel geometric framework for multimodal stress/pain detection utilizing Symmetric Positive Definite (SPD) matrices as a representation that incorporates the correlation relationship of physiological and behavioural signals from covariance and cross-covariance. Considering the non-linearity of the Riemannian manifold of SPD matrices, well-known machine learning techniques are not suited to classify these mat…
Swiss Federal Institute of Technology Lausanne（洛桑联邦理工学院）
Continuous and multimodal stress detection has been performed recently through wearable devices and machine learning algorithms. However, a well-known and important challenge of working on physiological signals recorded by conventional monitoring devices is missing data due to sensors insufficient contact and interference by other equipment. This challenge becomes more problematic when the user/patient is mentally or physically active or stressed because of more frequent conscious or subconscious movements. In this paper, we propose ReLearn, a robust machine learning framework for stress detec…
QualityFM: a Multimodal Physiological Signal Foundation Model with Self-Distillation for Signal Quality Challenges in Critically Ill Patients
Department of Electronics, Information and Bioengineering, Politecnico di Milano（电子、信息与生物工程系，米兰理工学院）；State Key Laboratory of Industrial Control Technology, Zhejiang University（工业控制技术国家重点实验室，浙江大学）
Tracing Human Stress from Physiological Signals using UWB Radar
Guangzhou University（广州大学）；Guangxi University（广西大学）；Fudan University（复旦大学）；Huazhong University of Science and Technology（华中科技大学）；Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；Singapore University of Technology and Design（新加坡科技设计大学）
UC, Davis Health（加州大学戴维斯健康中心）；University of California, Santa Barbara（加州大学圣巴巴拉分校）
Sepsis is a leading cause of death in the Intensive Care Units (ICU). Early detection of sepsis is critical for patient survival. In this paper, we propose a multimodal Transformer model for early sepsis prediction, using the physiological time series data and clinical notes for each patient within $36$ hours of ICU admission. Specifically, we aim to predict sepsis using only the first 12, 18, 24, 30 and 36 hours of laboratory measurements, vital signs, patient demographics, and clinical notes. We evaluate our model on two large critical care datasets: MIMIC-III and eICU-CRD. The proposed meth…
Physiological Signals are the most reliable form of signals for emotion recognition, as they cannot be controlled deliberately by the subject. Existing review papers on emotion recognition based on physiological signals surveyed only the regular steps involved in the workflow of emotion recognition such as preprocessing, feature extraction, and classification. While these are important steps, such steps are required for any signal processing application. Emotion recognition poses its own set of challenges that are very important to address for a robust system. Thus, to bridge the gap in the ex…
In this paper, we address the problem of multimodal emotion recognition from multiple physiological signals. We demonstrate that a Transformer-based approach is suitable for this task. In addition, we present how such models may be pretrained in a multimodal scenario to improve emotion recognition performances. We evaluate the benefits of using multimodal inputs and pre-training with our approach on a state-ofthe-art dataset.
HERMES: A Unified Open-Source Framework for Realtime Multimodal Physiological Sensing, Edge AI, and Intervention in Closed-Loop Smart Healthcare Applications
College of Information and Computer Sciences（信息与计算机科学学院）；University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）
Intensive Care Unit Electronic Health Records (ICU EHRs) store multimodal data about patients including clinical notes, sparse and irregularly sampled physiological time series, lab results, and more. To date, most methods designed to learn predictive models from ICU EHR data have focused on a single modality. In this paper, we leverage the recently proposed interpolation-prediction deep learning architecture(Shukla and Marlin 2019) as a basis for exploring how physiological time series data and clinical notes can be integrated into a unified mortality prediction model. We study both early and…
FedMultiEmo: Real-Time Emotion Recognition via Multimodal Federated Learning
Institute of Industrial Automation and Software Engineering, University of Stuttgart, Germany（工业自动化与软件工程研究所，斯图加特大学）；Electrical Engineering, University of Stuttgart, Germany（电气工程，斯图加特大学）；Graduate School of Excellence advanced Manufacturing Engineering, University of Stuttgart, Germany（先进制造工程优秀研究生院，斯图加特大学）
Physiological Information Reliability: Cross-Layer Adaptive Resource Allocation for Cardiovascular Sensing
Saveetha School of Engineering（萨维塔工程学院）；Government Kilpauk Medical College and Hospital（政府基尔帕医学院及医院）；SRM Institute of Science and Technology（SRM科学技术学院）
University of Augsburg（奥格斯堡大学）；University of Ulm（乌尔姆大学）；Imperial College London（帝国理工学院）
Emotion is an inherently subjective psychophysiological human-state and to produce an agreed-upon representation (gold standard) for continuous emotion requires a time-consuming and costly training procedure of multiple human annotators. There is strong evidence in the literature that physiological signals are sufficient objective markers for states of emotion, particularly arousal. In this contribution, we utilise a dataset which includes continuous emotion and physiological signals - Heartbeats per Minute (BPM), Electrodermal Activity (EDA), and Respiration-rate - captured during a stress in…
Northeastern University（东北大学）；University of Pittsburgh（匹兹堡大学）
There is an increasing consensus among re- searchers that making a computer emotionally intelligent with the ability to decode human affective states would allow a more meaningful and natural way of human-computer interactions (HCIs). One unobtrusive and non-invasive way of recognizing human affective states entails the exploration of how physiological signals vary under different emotional experiences. In particular, this paper explores the correlation between autonomically-mediated changes in multimodal body signals and discrete emotional states. In order to fully exploit the information in …
University of Cambridge（剑桥大学）；Alan Turing Institute（阿兰·图灵研究所）
Wearable devices such as smartwatches are becoming increasingly popular tools for objectively monitoring physical activity in free-living conditions. To date, research has primarily focused on the purely supervised task of human activity recognition, demonstrating limited success in inferring high-level health outcomes from low-level signals. Here, we present a novel self-supervised representation learning method using activity and heart rate (HR) signals without semantic labels. With a deep neural network, we set HR responses as the supervisory signal for the activity data, leveraging their u…
Towards Affordable, Non-Invasive Real-Time Hypoglycemia Detection Using Wearable Sensor Signals
Department of Computer Science, Old Dominion University, Norfolk, VA, USA（计算机科学系，旧 Dominion 大学）；Virginia Digital Maritime Center (VDMC), Old Dominion University, Norfolk, VA, USA（弗吉尼亚数字水路中心（VDMC），旧 Dominion 大学）
