Uni-modal identification systems are vulnerable to errors in sensor data collection and are therefore more likely to misidentify subjects. For instance, relying on data solely from an RGB face camera can cause problems in poorly lit environments or if subjects do not face the camera. Other identification methods such as electrocardiograms (ECG) have issues with improper lead connections to the skin. Errors in identification are minimized through the fusion of information gathered from both of these models. This paper proposes a methodology for combining the identification results of face and ECG data using Part A of the BioVid Heat Pain Database containing synchronized RGB-video and ECG data on 87 subjects. Using 10-fold cross-validation, face identification was 98.8% accurate, while the ECG identification was 96.1% accurate. By using a fusion approach the identification accuracy improved to 99.8%. Our proposed methodology allows for identification accuracies to be significantly improved by using disparate face and ECG models that have non-overlapping modalities.
An automatic classification method has been studied to effectively detect and recognize Electrocardiogram (ECG). Based on the synchronizing and orthogonal relationships of multiple leads, we propose a Multi-branch Convolution and Residual Network (MBCRNet) with three kinds of feature fusion methods for automatic detection of normal and abnormal ECG signals. Experiments are conducted on the Chinese Cardiovascular Disease Database (CCDD). Through 10-fold cross-validation, we achieve an average accuracy of 87.04% and a sensitivity of 89.93%, which outperforms previous methods under the same datab…
Stress analysis and assessment of affective states of mind using ECG as a physiological signal is a burning research topic in biomedical signal processing. However, existing literature provides only binary assessment of stress, while multiple levels of assessment may be more beneficial for healthcare applications. Furthermore, in present research, ECG signal for stress analysis is examined independently in spatial domain or in transform domains but the advantage of fusing these domains has not been fully utilized. To get the maximum advantage of fusing diferent domains, we introduce a dataset …
In this paper, we present a novel Image Fusion Model (IFM) for ECG heart-beat classification to overcome the weaknesses of existing machine learning techniques that rely either on manual feature extraction or direct utilization of 1D raw ECG signal. At the input of IFM, we first convert the heart beats of ECG into three different images using Gramian Angular Field (GAF), Recurrence Plot (RP) and Markov Transition Field (MTF) and then fuse these images to create a single imaging modality. We use AlexNet for feature extraction and classification and thus employ end to end deep learning. We perfo…
School of Information Science and Technology, University of Science and Technology of China（中国科学技术大学信息科学技术学院）；Department of Electrocardiogram, The First Affiliated Hospital of University of Science and Technology of China (Anhui Provincial Hospital)（中国科学技术大学附属第一医院（安徽省立医院）心电图科）
Automatic arrhythmia detection using 12-lead electrocardiogram (ECG) signal plays a critical role in early prevention and diagnosis of cardiovascular diseases. In the previous studies on automatic arrhythmia detection, most methods concatenated 12 leads of ECG into a matrix, and then input the matrix to a variety of feature extractors or deep neural networks for extracting useful information. Under such frameworks, these methods had the ability to extract comprehensive features (known as integrity) of 12-lead ECG since the information of each lead interacts with each other during training. How…
Multimodal Latent Fusion of ECG Leads for Early Assessment of Pulmonary Hypertension
School of Computer Science, University of Sheffield（谢菲尔德大学计算机科学学院）；Centre for Machine Intelligence, University of Sheffield（谢菲尔德大学智能中心）；Department of Electrical & Computer Science Engineering, IITRAM Ahmedabad（印度阿赫迈德亚布德IITRAM电子与计算机科学工程系）；Digital Environment Research Institute, Queen Mary University of London（伦敦玛丽女王大学数字环境研究院）；Nepal Applied Mathematics and Informatics Institute for research (NAAMII), Nepal（尼泊尔应用数学与信息技术研究所）；Department of Computing, Imperial College London（伦敦帝国学院计算机系）；School of Medicine and Population Health, University of Sheffield（谢菲尔德大学医学与人口健康学院）；Department of Clinical Radiology, Sheffield Teaching Hospitals（谢菲尔德教学医院放射科）；National Institute for Health and Care Research (NIHR), Sheffield Biomedical Research Centre（英国国家健康与护理研究所（NIHR）谢菲尔德生物医学研究中心）
Attention-Guided Fusion of 1D and 2D CNNs for Robust ECG-Based Biometric Recognition
PIMIS Laboratory, Electronics and Telecommunications Department（PIMIS实验室，电子与电信系）；Université du 8 Mai 1945（8月1945大学）；Electrical Engineering Department, University of 20 August 1955（电子工程系，20 August 1955大学）；Department of Electrical Engineering, Faculty of Science and Applied Sciences（电子工程系，科学与应用科学学院）；Larbi Ben M'hidi University（拉比·本·迈迪大学）；Department of Electronics and Communications, University of Larbi Tebessi（电子与通信系，拉比·塔贝西大学）
First-person interaction recognition is a challenging task because of unstable video conditions resulting from the camera wearer's movement. For human interaction recognition from a first-person viewpoint, this paper proposes a three-stream fusion network with two main parts: three-stream architecture and three-stream correlation fusion. Thre three-stream architecture captures the characteristics of the target appearance, target motion, and camera ego-motion. Meanwhile the three-stream correlation fusion combines the feature map of each of the three streams to consider the correlations among t…
Indian Institute of Technology Hyderabad（印度理工学院海得拉巴分校）
Objective: Heartbeat detection remains central to cardiac disease diagnosis and management, and is traditionally performed based on electrocardiogram (ECG). To improve robustness and accuracy of detection, especially, in certain critical-care scenarios, the use of additional physiological signals such as arterial blood pressure (BP) has recently been suggested. There, estimation of heartbeat location requires information fusion from multiple signals. However, reported efforts in this direction often obtain multimodal estimates somewhat indirectly, by voting among separately obtained signal-spe…
Johns Hopkins University（约翰斯·霍普金斯大学）；Hong Kong Baptist University（香港浸会大学）
This paper addresses fully automated multi-person tracking in complex environments with challenging occlusion and extensive pose variations. Our solution combines multiple detectors for a set of different regions of interest (e.g., full-body and head) for multi-person tracking. The use of multiple detectors leads to fewer miss detections as it is able to exploit the complementary strengths of the individual detectors. While the number of false positives may increase with the increased number of bounding boxes detected from multiple detectors, we propose to group the detection outputs by boundi…
Detecting people in images is a challenging problem. Differences in pose, clothing and lighting, along with other factors, cause a lot of variation in their appearance. To overcome these issues, we propose a system based on fused range and thermal infrared images. These measurements show considerably less variation and provide more meaningful information. We provide a brief introduction to the sensor technology used and propose a calibration method. Several data fusion algorithms are compared and their performance is assessed on a simulated data set. The results of initial experiments on real …
FreeLoRA: Enabling Training-Free LoRA Fusion for Autoregressive Multi-Subject Personalization
School of Artificial Intelligence, Jilin University（吉林大学人工智能学院）；Shanghai Innovation Institute（上海创新研究院）；School of Computer Science, Fudan University（复旦大学计算机学院）
Panasonic R&D Center（松下研发中心）；Panasonic Corporation（松下公司）；Core Element Technology Development Center（核心元件技术开发中心）
Person re-identification (re-ID) aims to accurately re- trieve a person from a large-scale database of images cap- tured across multiple cameras. Existing works learn deep representations using a large training subset of unique per- sons. However, identifying unseen persons is critical for a good re-ID algorithm. Moreover, the misalignment be- tween person crops to detection errors or pose variations leads to poor feature matching. In this work, we present a fusion of handcrafted features and deep feature representa- tion learned using multiple body parts to complement the global body features…
RGB-infrared person re-identification is a challenging task due to the intra-class variations and cross-modality discrepancy. Existing works mainly focus on learning modality-shared global representations by aligning image styles or feature distributions across modalities, while local feature from body part and relationships between person images are largely neglected. In this paper, we propose a Dual-level (i.e., local and global) Feature Fusion (DF^2) module by learning attention for discriminative feature from local to global manner. In particular, the attention for a local feature is deter…
University of Electronic Science and Technology of China（电子科技大学）；Shenzhen Institute for Advanced Study, UESTC（电子科技大学深圳高等研究院）
Unsupervised domain adaptive (UDA) person re-identification (ReID) has gained increasing attention for its effectiveness on the target domain without manual annotations. Most fine-tuning based UDA person ReID methods focus on encoding global features for pseudo labels generation, neglecting the local feature that can provide for the fine-grained information. To handle this issue, we propose a Learning Feature Fusion (LF2) framework for adaptively learning to fuse global and local features to obtain a more comprehensive fusion feature representation. Specifically, we first pre-train our model w…
Detecting and recognizing objects interacting with humans lie in the center of first-person (egocentric) daily activity recognition. However, due to noisy camera motion and frequent changes in viewpoint and scale, most of the previous egocentric action recognition methods fail to capture and model highly discriminative object features. In this work, we propose a novel pipeline for first-person daily activity recognition, aiming at more discriminative object feature representation and object-motion feature fusion. Our object feature extraction and representation pipeline is inspired by the rece…
SenseTime Group Limited（商汤科技集团有限公司）；The Chinese University of HongKong（香港中文大学）
Although person re-identification (ReID) has achieved significant improvement recently by enforcing part alignment, it is still a challenging task when it comes to distinguishing visually similar identities or identifying the occluded person. In these scenarios, magnifying details in each part features and selectively fusing them together may provide a feasible solution. In this work, we propose MagnifierNet, a triple-branch network which accurately mines details from whole to parts. Firstly, the holistic salient features are encoded by a global branch. Secondly, to enhance detailed representa…
Person re-identification plays a significant role in realistic scenarios due to its various applications in public security and video surveillance. Recently, leveraging the supervised or semi-unsupervised learning paradigms, which benefits from the large-scale datasets and strong computing performance, has achieved a competitive performance on a specific target domain. However, when Re-ID models are directly deployed in a new domain without target samples, they always suffer from considerable performance degradation and poor domain generalization. To address this challenge, we propose a Deep M…
Beijing University of Posts and Telecommucations（北京邮电大学）
Video-based person recognition is challenging due to persons being blocked and blurred, and the variation of shooting angle. Previous research always focused on person recognition on still images, ignoring similarity and continuity between video frames. To tackle the challenges above, we propose a novel Frame Aggregation and Multi-Modal Fusion (FAMF) framework for video-based person recognition, which aggregates face features and incorporates them with multi-modal information to identify persons in videos. For frame aggregation, we propose a novel trainable layer based on NetVLAD (named Attent…
Utilizing the sensor characteristics of the audio, visible camera, and thermal camera, the robustness of person recognition can be enhanced. Existing multimodal person recognition frameworks are primarily formulated assuming that multimodal data is always available. In this paper, we propose a novel trimodal sensor fusion framework using the audio, visible, and thermal camera, which addresses the missing modality problem. In the framework, a novel deep latent embedding framework, termed the AVTNet, is proposed to learn multiple latent embeddings. Also, a novel loss function, termed missing mod…
MIT Computer Science and Artificial Intelligence Laboratory（麻省理工学院计算机科学与人工智能实验室）
In this paper, we present a multi-modal online person verification system using both speech and visual signals. Inspired by neuroscientific findings on the association of voice and face, we propose an attention-based end-to-end neural network that learns multi-sensory associations for the task of person verification. The attention mechanism in our proposed network learns to conditionally select a salient modality between speech and facial representations that provides a balance between complementary inputs. By virtue of this capability, the network is robust to missing or corrupted data from e…
Tencent Youtu Lab（腾讯优图实验室）；Sun Yat-sen University（中山大学）；Southern University of Science and Technology（南方科技大学）
Recently, the research interest of person re-identification (ReID) has gradually turned to video-based methods, which acquire a person representation by aggregating frame features of an entire video. However, existing video-based ReID methods do not consider the semantic difference brought by the outputs of different network stages, which potentially compromises the information richness of the person features. Furthermore, traditional methods ignore important relationship among frames, which causes information redundancy in fusion along the time axis. To address these issues, we propose a nove…
Person identification (P-ID) under real unconstrained noisy environments is a huge challenge. In multiple-feature learning with Deep Convolutional Neural Networks (DCNNs) or Machine Learning method for large-scale person identification in the wild, the key is to design an appropriate strategy for decision layer fusion or feature layer fusion which can enhance discriminative power. It is necessary to extract different types of valid features and establish a reasonable framework to fuse different types of information. In traditional methods, different persons are identified based on single modal…
Personalized Federated Learning via Dual-Prompt Optimization and Cross Fusion
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Hamad Bin Khalifa University（哈马德·本·卡西姆大学）；Central South University（中南大学）
Unity is Strength: Unifying Convolutional and Transformeral Features for Better Person Re-Identification
School of Artificial Intelligence, Dalian University of Technology（大连理工大学人工智能学院）；School of Computer Science and Artificial Intelligence, Wuhan University of Technology（武汉理工大学计算机科学与人工智能学院）；School of Computer Science and Technology, Anhui University（安徽大学计算机科学与技术学院）；School of Information and Communication Engineering, Dalian University of Technology（大连理工大学信息与通信工程学院）
Spectral-Temporal Fusion Representation for Person-in-Bed Detection
Group of Intelligent Signal Processing (GISP), College of Computer Science and Technology, Harbin Engineering University（智能信号处理组，计算机科学与技术学院，哈尔滨工程大学）；Acoustics Lab, University of Technology Sydney（声学实验室，悉尼技术大学）
