This paper presents a novel approach to processing multimodal data for dynamic emotion recognition, named as the Multimodal Masked Autoencoder for Dynamic Emotion Recognition (MultiMAE-DER). The MultiMAE-DER leverages the closely correlated representation information within spatiotemporal sequences across visual and audio modalities. By utilizing a pre-trained masked autoencoder model, the MultiMAEDER is accomplished through simple, straightforward finetuning. The performance of the MultiMAE-DER is enhanced by optimizing six fusion strategies for multimodal input sequences. These strategies address dynamic feature correlations within cross-domain data across spatial, temporal, and spatiotemporal sequences. In comparison to state-of-the-art multimodal supervised learning models for dynamic emotion recognition, MultiMAE-DER enhances the weighted average recall (WAR) by 4.41% on the RAVDESS dataset and by 2.06% on the CREMAD. Furthermore, when compared with the state-of-the-art model of multimodal self-supervised learning, MultiMAE-DER achieves a 1.86% higher WAR on the IEMOCAP dataset.
Swiss Federal Institute of Technology Lausanne (EPFL)（洛桑联邦理工学院）
We propose a pre-training strategy called Multi-modal Multi-task Masked Autoencoders (MultiMAE). It differs from standard Masked Autoencoding in two key aspects: I) it can optionally accept additional modalities of information in the input besides the RGB image (hence "multi-modal"), and II) its training objective accordingly includes predicting multiple outputs besides the RGB image (hence "multi-task"). We make use of masking (across image patches and input modalities) to make training MultiMAE tractable as well as to ensure cross-modality predictive coding is indeed learned by the network. …
MultiMAE for Brain MRIs: Robustness to Missing Inputs Using Multi-Modal Masked Autoencoder
Department of Radiation Oncology, TUM University Hospital, Munich, Germany（辐射肿瘤科，技术大学慕尼黑医院，慕尼黑，德国）；Chair for AI in Healthcare and Medicine, Technical University of Munich (TUM)（医疗与医学人工智能教研室，技术大学慕尼黑（TUM））；TUM University Hospital, Munich, Germany（技术大学慕尼黑医院，慕尼黑，德国）；Chair for AI for Image-Guided Diagnosis and Therapy, Technical University of Munich (TUM)（图像引导诊断与治疗人工智能教研室，技术大学慕尼黑（TUM））；Munich Center for Machine Learning (MCML), Munich, Germany（慕尼黑机器学习中心（MCML），慕尼黑，德国）；Department of Computing, Imperial College London, London, UK（计算系，伦敦帝国学院，伦敦，英国）；Deutsches Konsortium für Translationale Krebsforschung (DKTK), Partner Site Munich, Munich, Germany（德国转化癌症研究联盟（DKTK），慕尼黑合作伙伴站点，慕尼黑，德国）；Institute of Radiation Medicine (IRM), Department of Radiation Sciences (DRS), Helmholtz Center Munich, Munich, Germany（辐射医学研究所（IRM），辐射科学部门（DRS），慕尼黑海德堡中心，慕尼黑，德国）
Multi-Masked Querying Network for Robust Emotion Recognition from Incomplete Multi-Modal Physiological Signals
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518055, China（深圳先进技术研究院，中国科学院，深圳518055，中国）；Southern University of Science and Technology, Shenzhen 518055, China（南方科技大学，深圳518055，中国）
HiCMAE: Hierarchical Contrastive Masked Autoencoder for Self-Supervised Audio-Visual Emotion Recognition
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Department of Automation, Tsinghua University（清华大学自动化系）；Beijing National Research Center for Information Science and Technology, Tsinghua University（清华大学北京信息科学与技术国家研究中心）
Renmin University of China（中国人民大学）；National University of Singapore（新加坡国立大学）
Multimodal emotion recognition study is hindered by the lack of labelled corpora in terms of scale and diversity, due to the high annotation cost and label ambiguity. In this paper, we propose a pre-training model \textbf{MEmoBERT} for multimodal emotion recognition, which learns multimodal joint representations through self-supervised learning from large-scale unlabeled video data that come in sheer volume. Furthermore, unlike the conventional "pre-train, finetune" paradigm, we propose a prompt-based method that reformulates the downstream emotion classification task as a masked text predicti…
Mamba-VA: A Mamba-based Approach for Continuous Emotion Recognition in Valence-Arousal Space
Nanjing University of Posts and Telecommunications（南京邮电大学）；Nanjing University of Science and Technology（南京理工大学）；ZiJin College（紫金学院）；School of Computer and Artificial Intelligence（计算机与人工智能学院）
本文提出 Mamba-VA，结合 MAE、TCN 与 Mamba 建模视频情绪的局部和长程时间动态，在 ABAW VA 估计任务上优于基线。
Social-MAE: A Transformer-Based Multimodal Autoencoder for Face and Voice
Numediart Institute, ISIA Lab, University of Mons（Numediart研究院、ISIA实验室、蒙斯大学）；Institute for Creative Technologies, University of Southern California（创意技术研究所、南加州大学）
Scalable Audio-Visual Masked Autoencoders for Efficient Affective Video Facial Analysis
School of Computer Science and Technology, Xi’an Jiaotong University（西安交通大学计算机科学与技术学院）；School of Cyber Science and Engineering, Zhengzhou University（郑州大学网络科学与工程学院）；School of Software, Northwestern Polytechnical University（西北工业大学软件学院）；Institute of Advanced Technology, University of Science and Technology of China（中国科学技术大学先进技术研究院）；Inspur Group（Inspur集团）；Research Center for Space Computing System, Zhejiang Lab（浙江实验室空间计算系统研究中心）
针对情感视频面部分析数据有限及跨模态建模难题，提出可扩展音视频 MAE 模型 AVF-MAE++，通过双重掩码、相关性学习模块和三阶段训练，在17个数据集上取得SOTA。
Artificially intelligent perception is increasingly present in the lives of every one of us. Vehicles are no exception, (...) In the near future, pattern recognition will have an even stronger role in vehicles, as self-driving cars will require automated ways to understand what is happening around (and within) them and act accordingly. (...) This doctoral work focused on advancing in-vehicle sensing through the research of novel computer vision and pattern recognition methodologies for both biometrics and wellbeing monitoring. The main focus has been on electrocardiogram (ECG) biometrics, a tr…
