Multimodal deep learning has substantially improved electrocardiogram (ECG) classification by jointly leveraging time, frequency, and time-frequency representations. However, existing generative models typically synthesize these modalities independently, resulting in synthetic ECG data that are visually plausible yet physiologically inconsistent across domains. This work establishes a Complementarity-Preserving Generative Theory (CPGT), which posits that physiologically valid multimodal signal generation requires explicit preservation of cross-domain complementarity rather than loosely coupled modality synthesis. We instantiate CPGT through Q-CFD-GAN, a quantum-inspired generative framework that models multimodal ECG structure within a complex-valued latent space and enforces complementarity-aware constraints regulating mutual information, redundancy, and morphological coherence. Experimental evaluation demonstrates that Q-CFD-GAN reduces latent embedding variance by 82%, decreases classifier-based plausibility error by 26.6%, and restores tri-domain complementarity from 0.56 to 0.91, while achieving the lowest observed morphology deviation (3.8%). These findings show that preserving multimodal information geometry, rather than optimizing modality-specific fidelity alone, is essential for generating synthetic ECG signals that remain physiologically meaningful and suitable for downstream clinical machine-learning applications.
MM-DADM: Multimodal Drug-Aware Diffusion Model for Virtual Clinical Trials
Zhejiang University（浙江大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
CardioRAG: A Retrieval-Augmented Generation Framework for Multimodal Chagas Disease Detection
Department of Electrical and Electronic Engineering Imperial College London（帝国理工学院电子与电气工程系）；Department of Civil and Environmental Engineering Imperial College London（帝国理工学院土木与环境工程系）；Institute of Biomedical Engineering Department of Engineering Science University of Oxford（牛津大学生物医学工程研究所）
MEETI: A Multimodal ECG Dataset from MIMIC-IV-ECG with Signals, Images, Features and Interpretations
HeartVoice Medical Technology（HeartVoice医疗科技）；Saw Swee Hock School of Public Health and Institute of Data Science（Saw Swee Hock公共卫生学院和数据科学研究所）；National University of Singapore（新加坡国立大学）；Department of Cardiology, Peking University People’s Hospital（北京大学人民医院心内科）；College of Integrative Chinese and Western Medicine, Anhui University of Chinese Medicine（安徽中医药大学整合中西医学学院）；National Institute of Health Data Science, Peking University（北京大学国家健康数据科学研究院）；Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）
Chonnam National University（全南国立大学）；University of Arkansas（阿肯色大学）；West Virginia University（西弗吉尼亚大学）
Electrocardiogram (ECG) signal is one of the most effective sources of information mainly employed for the diagnosis and prediction of cardiovascular diseases (CVDs) connected with the abnormalities in heart rhythm. Clearly, single modality ECG (i.e. time series) cannot convey its complete characteristics, thus, exploiting both time and time-frequency modalities in the form of time-series data and spectrogram is needed. Leveraging the cutting-edge self-supervised learning (SSL) technique on unlabeled data, we propose SSL-based multimodality ECG classification. Our proposed network follows SSL …
Electrocardiogram (ECG) is an authoritative source to diagnose and counter critical cardiovascular syndromes such as arrhythmia and myocardial infarction (MI). Current machine learning techniques either depend on manually extracted features or large and complex deep learning networks which merely utilize the 1D ECG signal directly. Since intelligent multimodal fusion can perform at the stateof-the-art level with an efficient deep network, therefore, in this paper, we propose two computationally efficient multimodal fusion frameworks for ECG heart beat classification called Multimodal Image Fus…
In this paper, we present a novel Image Fusion Model (IFM) for ECG heart-beat classification to overcome the weaknesses of existing machine learning techniques that rely either on manual feature extraction or direct utilization of 1D raw ECG signal. At the input of IFM, we first convert the heart beats of ECG into three different images using Gramian Angular Field (GAF), Recurrence Plot (RP) and Markov Transition Field (MTF) and then fuse these images to create a single imaging modality. We use AlexNet for feature extraction and classification and thus employ end to end deep learning. We perfo…
MEIT: Multimodal Electrocardiogram Instruction Tuning on Large Language Models for Report Generation
The Ohio State University（俄亥俄州立大学）；Imperial College London（伦敦帝国理工学院）；The University of Hong Kong（香港大学）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）
TSRNet: Simple Framework for Real-time ECG Anomaly Detection with Multimodal Time and Spectrogram Restoration Network
University of Arkansas（阿肯色大学）；University of Science, Vietnam National University（越南国家大学科学大学）；John von Neumann Institute, Vietnam National University（越南国家大学约翰·冯·诺依曼研究所）；West Virginia University（西弗吉尼亚大学）
Method: In this study, a new method is introduced for distinguishing noise-free segments of ECG from noisy segments that use sample amplitude dispersion with an adoptive threshold for variance of samples amplitude and a method which uses compatibility of detected beats in ECG and some of the other signals which are related to the heart activity such as BP, arterial pressure (ART) and pulmonary artery pressure (PAP). A prioritization is applied in other pulsatile signals based on the amplitude and clarity of the peaks on them, and a fusion strategy is employed for segments on which ECG is noisy…
Scaling to Multimodal and Multichannel Heart Sound Classification with Synthetic and Augmented Biosignals
School of Electrical Engineering, Computing, and Mathematical Sciences (EECMS), Faculty of Science and Engineering, Curtin University, Bentley, WA 6102, Australia（电气工程、计算与数学科学学院（EECMS），科学与工程学院， Curtin 大学，Bentley，WA 6102，澳大利亚）
ECG is an attractive option to assess stress in serious Virtual Reality (VR) applications due to its non-invasive nature. However, the existing Machine Learning (ML) models perform poorly. Moreover, existing studies only perform a binary stress assessment, while to develop a more engaging biofeedback-based application, multi-level assessment is necessary. Existing studies annotate and classify a single experience (e.g. watching a VR video) to a single stress level, which again prevents design of dynamic experiences where real-time in-game stress assessment can be utilized. In this paper, we re…
PAMF: Prior-Aware Multimodal Fusion for Incomplete Time Series Data
Department of Computer Science, University of Central Florida（中央佛罗里达大学计算机科学系）；Department of Computer Science, University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校计算机科学系）
University of Chinese Academy of Sciences（中国科学院大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Noah’s Ark Lab（诺亚方舟实验室）；The University of Texas at San Antonio（德克萨斯大学圣安东尼奥分校）
Multimodal learning aims to discover the relationship between multiple modalities. It has become an important research topic due to extensive multimodal applications such as cross-modal retrieval. This paper attempts to address the modality heterogeneity problem based on Gaussian process latent variable models (GPLVMs) to represent multimodal data in a common space. Previous multimodal GPLVM extensions generally adopt individual learning schemes on latent representations and kernel hyperparameters, which ignore their intrinsic relationship. To exploit strong complementarity among different mod…
Multimodal Latent Fusion of ECG Leads for Early Assessment of Pulmonary Hypertension
School of Computer Science, University of Sheffield（谢菲尔德大学计算机科学学院）；Centre for Machine Intelligence, University of Sheffield（谢菲尔德大学智能中心）；Department of Electrical & Computer Science Engineering, IITRAM Ahmedabad（印度阿赫迈德亚布德IITRAM电子与计算机科学工程系）；Digital Environment Research Institute, Queen Mary University of London（伦敦玛丽女王大学数字环境研究院）；Nepal Applied Mathematics and Informatics Institute for research (NAAMII), Nepal（尼泊尔应用数学与信息技术研究所）；Department of Computing, Imperial College London（伦敦帝国学院计算机系）；School of Medicine and Population Health, University of Sheffield（谢菲尔德大学医学与人口健康学院）；Department of Clinical Radiology, Sheffield Teaching Hospitals（谢菲尔德教学医院放射科）；National Institute for Health and Care Research (NIHR), Sheffield Biomedical Research Centre（英国国家健康与护理研究所（NIHR）谢菲尔德生物医学研究中心）
Interpretable Multimodal EEG-ECG Framework for Neonatal Seizure Prediction and Patient-Adaptive Transfer Learning
Department of Electronic and Telecommunication Engineering, University of Moratuwa, Sri Lanka（电子与电信工程系，乌尔塔瓦大学，斯里兰卡）；Department of Biomedical Engineering, University of Melbourne, Australia（生物医学工程系，墨尔本大学，澳大利亚）；Department of Electrical and Computer Engineering, University of British Columbia, Canada（电气与计算机工程系，不列颠哥伦比亚大学，加拿大）；Department of Pediatrics, Faculty of Medicine, University of Colombo, Sri Lanka（儿科系，医学院，科伦坡大学，斯里兰卡）
A Multimodal Deep Learning Framework for Predicting ICU Deterioration: Integrating ECG Waveforms with Clinical Data and Clinician Benchmarking
AI4Health Division, Carl von Ossietzky Universität Oldenburg（AI4Health部门，奥尔登堡大学）；CAPASITS, Secretariat of Health of the State of Colima（CAPASITS，科利马州卫生秘书处）；Department of Interventional Cardiology, Hospital Puerta de Hierro Colima（介入心脏病学部，科利马医院）；Institute for Medical Informatics and Statistics, Kiel University and University Hospital Schleswig-Holstein（医学信息学与统计研究所，基尔大学和石勒苏益格-荷尔斯泰因大学医院）；Department of Anesthesiology, Intensive Care Medicine, Emergency Medicine, and Pain Therapy, Carl von Ossietzky Universität & Klinikum Oldenburg（麻醉学、重症医学、急诊医学和疼痛治疗部，奥尔登堡大学及奥尔登堡诊所）；Department of Cardiology, Angiology and Intensive Care Medicine, Charité Campus Mitte, German Heart Center of the Charité-University Medicine Berlin（心内科、血管科和重症医学部，Charité梅尔特校区，Charité-大学医学中心柏林德国心脏中心）
Cross-Learning Between ECG and PCG: Exploring Common and Exclusive Characteristics of Bimodal Electromechanical Cardiac Waveforms
School of Medicine, Emory University（埃默里大学医学院）；Rollins School of Public Health, Department of Epidemiology, Emory University（埃默里大学罗林斯公共卫生学院流行病学系）；Department of Biomedical Engineering, Georgia Institute of Technology and Emory University（佐治亚理工学院生物医学工程系）
Queen’s University（女王大学）；Ingenuity Labs Research Institute（创新实验室研究所）
Classification of human emotions can play an essential role in the design and improvement of human-machine systems. While individual biological signals such as Electrocardiogram (ECG) and Electrodermal Activity (EDA) have been widely used for emotion recognition with machine learning methods, multimodal approaches generally fuse extracted features or final classification/regression results to boost performance. To enhance multimodal learning, we present a novel attentive cross-modal connection to share information between convolutional neural networks responsible for learning individual modali…
Indian Institute of Technology Hyderabad（印度理工学院海得拉巴分校）
Objective: Heartbeat detection remains central to cardiac disease diagnosis and management, and is traditionally performed based on electrocardiogram (ECG). To improve robustness and accuracy of detection, especially, in certain critical-care scenarios, the use of additional physiological signals such as arterial blood pressure (BP) has recently been suggested. There, estimation of heartbeat location requires information fusion from multiple signals. However, reported efforts in this direction often obtain multimodal estimates somewhat indirectly, by voting among separately obtained signal-spe…
Detection of Autonomic Dysreflexia in Individuals With Spinal Cord Injury Using Multimodal Wearable Sensors
SCAI-Lab, Department of Health Sciences and Technology (D-HEST), ETH Zurich（SCAI实验室，健康科学与技术系（D-HEST），苏黎世联邦理工学院）；School of Computation, Information and Technology, Technical University of Munich（计算、信息与技术学院，慕尼黑技术大学）；Swiss Paraplegic Research (SPF)（瑞士瘫痪研究机构（SPF））；Institute of Advanced Production Technologies, University of Valladolid（先进生产技术研究所，瓦尔达利德大学）；Swiss Paraplegic Centre (SPZ)（瑞士瘫痪中心（SPZ））；Balgrist University Hospital（巴尔格里斯大学医院）
