Emotion recognition in real-world environments is hindered by partial occlusions, missing modalities, and severe class imbalance. To address these issues, particularly for the Affective Behavior Analysis in-the-wild (ABAW) Expression challenge, we propose a multimodal framework that dynamically fuses visual and audio representations. Our approach uses a dual-branch Transformer architecture featuring a safe cross-attention mechanism and a modality dropout strategy. This design allows the network to rely on audio-based predictions when visual cues are absent. To mitigate the long-tail distribution of the Aff-Wild2 dataset, we apply focal loss optimization, combined with a sliding-window soft voting strategy to capture dynamic emotional transitions and reduce frame-level classification jitter. Experiments demonstrate that our framework effectively handles missing modalities and complex spatiotemporal dependencies, achieving an accuracy of 60.79% and an F1-score of 0.5029 on the Aff-Wild2 validation set.
Team RAS in 11th ABAW Competition: Multimodal Ambivalence Recognition Approach
St. Petersburg Federal Research Center of the Russian Academy of Sciences (SPC RAS)（俄罗斯科学院圣彼得堡联邦研究中心）；HSE University（圣彼得堡高等经济学院）；ITMO University（圣彼得堡国立信息技术机械与光学大学）
With the assumption that a video dataset is multimodality annotated in which auditory and visual modalities both are labeled or class-relevant, current multimodal methods apply modality fusion or cross-modality attention. However, effectively leveraging the audio modality in vision-specific annotated videos for action recognition is of particular challenge. To tackle this challenge, we propose a novel audio-visual framework that effectively leverages the audio modality in any solely vision-specific annotated dataset. We adopt the language models (e.g., BERT) to build a semantic audio-video lab…
Backbone-Conditional Behavior of Modality Gating in Multi-Modal Prostate MRI Segmentation: A 5-Fold Cross-Validation and Gate Mechanism Analysis
The Second Xiangya Hospital of Central South University（中南大学湘雅医学院第二医院）；The Third Xiangya Hospital of Central South University（中南大学湘雅医学院第三医院）；School of Life Sciences, Central South University（中南大学生命科学学院）；Hunan Provincial Key Laboratory of Medical Information Research (Central South University)（湖南省医学信息研究重点实验室（中南大学））；Hunan Provincial Clinical Medical Research Center for Cardiovascular Intelligent Medicine（湖南省心血管智能医学临床医学研究中心）
MGPC: Multimodal Network for Generalizable Point Cloud Completion With Modality Dropout and Progressive Decoding
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation of Chinese Academy of Sciences（中国科学院自动化研究所 multimodal 人工智能系统国家重点实验室）；Chemical Defense Institute, Academy of Military Sciences（军事科学院化学防御研究院）；Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation of Chinese Academy of Sciences（中国科学院自动化研究所 复杂系统认知与决策智能重点实验室）
Elekta（医科达）；KTH - Royal institute of Technology（KTH皇家理工学院）
Over the last few years machine learning has demonstrated groundbreaking results in many areas of medical image analysis, including segmentation. A key assumption, however, is that the train- and test distributions match. We study a realistic scenario where this assumption is clearly violated, namely segmentation with missing input modalities. We describe two neural network approaches that can handle a variable number of input modalities. The first is modality dropout: a simple but surprisingly effective modification of the training. The second is the unified representation network: a network …
Modality Discrepancy Transformer for Ambivalence and Hesitancy Recognition
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Future Technology, University of Chinese Academy of Sciences（中国科学院大学未来技术学院）；College of Computer and Information Engineering, Tianjin Normal University（天津师范大学计算机与信息工程学院）
CCSD: Cross-Modal Compositional Self-Distillation for Robust Brain Tumor Segmentation with Missing Modalities
Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University（上海智能自主系统研究院，同济大学）；College of Biomedical Engineering, Fudan University（生物医学工程学院，复旦大学）；School of Medicine, Tongji University（医学院，同济大学）；College of Information Engineering, China Jiliang University（信息工程学院，中国嘉兴大学）
While audio-visual speech models can yield superior performance and robustness compared to audio-only models, their development and adoption are hindered by the lack of labeled and unlabeled audio-visual data and the cost to deploy one model per modality. In this paper, we present u-HuBERT, a self-supervised pre-training framework that can leverage both multimodal and unimodal speech with a unified masked cluster prediction objective. By utilizing modality dropout during pre-training, we demonstrate that a single fine-tuned model can achieve performance on par or better than the state-of-the-a…
Robust Multispectral Semantic Segmentation under Missing or Full Modalities via Structured Latent Projection
Department of Computer Engineering, Ankara University（安卡拉大学计算机工程系）；Department of Modeling and Simulation, Graduate School of Informatics, METU（信息学院建模与模拟系，METU）
No Modality Left Behind: Dynamic Model Generation for Incomplete Medical Data
Computational Imaging Research Lab（计算成像研究实验室）；Department for Biomedical Imaging and Image-guided Therapy（生物医学成像与影像引导治疗部门）；Medical University of Vienna（维也纳医学大学）；Comprehensive Center for Artificial Intelligence in Medicine（医学人工智能综合中心）；Christian Doppler Laboratory for Machine Learning Driven Precision Imaging（机器学习驱动精准成像的克里斯蒂安·多普勒实验室）；Department of Biomedical Imaging and Image-guided Therapy（生物医学成像与影像引导治疗部门）；Athinoula A. Martinos Center for Biomedical Imaging（阿提诺拉·A·马丁努斯生物医学成像中心）；Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）；Department of Radiology（放射科）；Division of General and Pediatric Radiology（普通和儿童放射科部门）
Nanyang Technological University（南洋理工大学）；Huazhong University of Science and Technology（华中科技大学）
We propose an audio-visual spatial-temporal deep neural network with: (1) a visual block containing a pretrained 2D-CNN followed by a temporal convolutional network (TCN); (2) an aural block containing several parallel TCNs; and (3) a leader-follower attentive fusion block combining the audio-visual information. The TCN with large history coverage enables our model to exploit spatial-temporal information within a much larger window length (i.e., 300) than that from the baseline and state-of-the-art methods (i.e., 36 or 48). The fusion block emphasizes the visual modality while exploits the noi…
Tampere University（坦佩雷大学）；Aarhus University（奥胡斯大学）
In this paper, we consider the problem of multimodal data analysis with a use case of audiovisual emotion recognition. We propose an architecture capable of learning from raw data and describe three variants of it with distinct modality fusion mechanisms. While most of the previous works consider the ideal scenario of presence of both modalities at all times during inference, we evaluate the robustness of the model in the unconstrained settings where one modality is absent or noisy, and propose a method to mitigate these limitations in a form of modality dropout. Most importantly, we find that…
University at Buffalo, The State University of New York（纽约州立大学布法罗分校）
Brain Tumor Segmentation from magnetic resonance imaging (MRI) is a critical technique for early diagnosis. However, rather than having complete four modalities as in BraTS dataset, it is common to have missing modalities in clinical scenarios. We design a brain tumor segmentation algorithm that is robust to the absence of any modality. Our network includes a channel-independent encoding path and a feature-fusion decoding path. We use self-supervised training through channel dropout and also propose a novel domain adaptation method on feature maps to recover the information from the missing ch…
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）
The Visual Dialog task requires a model to exploit both image and conversational context information to generate the next response to the dialogue. However, via manual analysis, we find that a large number of conversational questions can be answered by only looking at the image without any access to the context history, while others still need the conversation context to predict the correct answers. We demonstrate that due to this reason, previous joint-modality (history and image) models over-rely on and are more prone to memorizing the dialogue history (e.g., by extracting certain keywords o…
Latent Space Guided Scenario Sampling for Multimodal Segmentation Under Missing Modalities
organization= Department of Computer Engineering, Ankara University, Ankara, T \"u rkiye；organization= Department of Modeling；Simulation, Graduate School of Informatics, METU, Ankara, T \"u rkiye
MacJEPA: Missingness-Robust Audio-Visual Recognition from Untrimmed Egocentric Videos
Peter L. Reichertz Institute for Medical Informatics, Hannover Medical School（汉诺威医学院彼得·L·赖歇茨医学信息学研究所）；Lower Saxony Center for Artificial Intelligence and Causal Methods in Medicine (CAIMed)（下萨克森州医学人工智能与因果方法中心（CAIMed））
Multiple Sclerosis (MS) is a chronic neuroinflammatory disease and multi-modality MRIs are routinely used to monitor MS lesions. Many automatic MS lesion segmentation models have been developed and have reached human-level performance. However, most established methods assume the MRI modalities used during training are also available during testing, which is not guaranteed in clinical practice. Previously, a training strategy termed Modality Dropout (ModDrop) has been applied to MS lesion segmentation to achieve the state-of-the-art performance with missing modality. In this paper, we present …
Resilient Vision-Tabular Multimodal Learning under Modality Missingness
Research Unit of Artificial Intelligence and Computer Systems, Department of Engineering, Università Campus Bio-Medico di Roma（人工智能与计算机系统研究单位，工程系，罗马生物医学大学）；Department of Diagnostics and Intervention, Radiation Physics, Biomedical Engineering, Umeå University（诊断与干预系，辐射物理，生物医学工程，乌梅大学）
DropMAE: Learning Representations via Masked Autoencoders with Spatial-Attention Dropout for Temporal Matching Tasks
City University of Hong Kong（香港城市大学）；Alibaba DAMO Academy（阿里巴巴达摩院）；Queen Mary University of London（伦敦玛丽女王大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
Human affective recognition is an important factor in human-computer interaction. However, the method development with in-the-wild data is not yet accurate enough for practical usage. In this paper, we introduce the affective recognition method focusing on valence-arousal (VA) and expression (EXP) that was submitted to the Affective Behavior Analysis in-the-wild (ABAW) 2020 Contest. Since we considered that affective behaviors have many observable features that have their own time frames, we introduced multiple optimized time windows (short-term, middle-term, and long-term) into our analyzing …
In this work we tackle the task of video-based audio-visual emotion recognition, within the premises of the 2nd Workshop and Competition on Affective Behavior Analysis in-the-wild (ABAW2). Poor illumination conditions, head/body orientation and low image resolution constitute factors that can potentially hinder performance in case of methodologies that solely rely on the extraction and analysis of facial features. In order to alleviate this problem, we leverage both bodily and contextual features, as part of a broader emotion recognition framework. We choose to use a standard CNN-RNN cascade a…
eHealth Center, Faculty of Computer Science, Multimedia and Telecommunicactions, Universitat Oberta de Catalunya（加泰罗尼亚开放大学计算机科学、多媒体与电信学院电子健康中心）；MIND/IN2UB, Department of Electronic and Biomedical Engineeering, Universitat de Barcelona（巴塞罗那大学电子与生物医学工程系MIND/IN2UB）；Institute of Semiconductor Technology (IHT) & Laboratory for Emerging Nanometrology (LENA), Technische Universität Braunschweig（布伦瑞克工业大学半导体技术研究所（IHT）和新兴纳米计量实验室（LENA））
Human activity recognition based on video streams has received numerous attentions in recent years. Due to lack of depth information, RGB video based activity recognition performs poorly compared to RGB-D video based solutions. On the other hand, acquiring depth information, inertia etc. is costly and requires special equipment, whereas RGB video streams are available in ordinary cameras. Hence, our goal is to investigate whether similar or even higher accuracy can be achieved with RGB-only modality. In this regard, we propose a novel framework that couples skeleton data extracted from RGB vid…
Cascaded Multi-Modal Mixing Transformers for Alzheimer's Disease Classification with Incomplete Data
Queensland Brain Institute, The University of Queensland（昆士兰脑研究所，昆士兰大学）；School of Information Technology and Electrical Engineering, The University of Queensland（信息技术与电气工程学院，昆士兰大学）
Conflict-Aware Multimodal Fusion for Ambivalence and Hesitancy Recognition
University of the Basque Country UPV/EHU（巴斯克大学UPV/EHU）；Laboratory of LESIA, University of Biskra（贝斯克拉大学LESIA实验室）；Lab. de Génie Electrique (LAGE), University Kasdi Merbah Ouargla（奥尔加拉大学LAGE实验室）；Institute of Electronics, Microelectronics and Nanotechnology (IEMN), Polytechnic University of Hauts-de-France, University of Lille（电子、微电子与纳米技术研究所（IEMN），法国 Hauts-de-France 工业大学，里尔大学）；Sorbonne Center for Artificial Intelligence, Sorbonne University Abu Dhabi, UAE（索邦人工智能中心，索邦大学阿布扎比校区，阿联酋）
Robust 3D object detection is critical for safe autonomous driving. Camera and radar sensors are synergistic as they capture complementary information and work well under different environmental conditions. Fusing camera and radar data is challenging, however, as each of the sensors lacks information along a perpendicular axis, that is, depth is unknown to camera and elevation is unknown to radar. We propose the camera-radar matching network CramNet, an efficient approach to fuse the sensor readings from camera and radar in a joint 3D space. To leverage radar range measurements for better came…
