Multimodal perception improves autonomous-driving performance by combining complementary sensor information. However, stronger multimodal predictions do not necessarily imply stronger representations in each modality-specific branch. Such branch-level weaknesses can remain hidden in conventional output-level evaluation. We introduce UBD-Bench, a systematic benchmark for studying unimodal branch degradation (UBD) in multimodal perception. UBD-Bench uses a Paired Frozen-Branch Probe to compare the task-relevant representation quality of the same modality after independent unimodal training and multimodal joint training under matched decoding conditions. Across four autonomous-driving perception tasks and 10 models, we find that Camera branches consistently degrade after multimodal joint training, whereas LiDAR branches exhibit weaker and more model-dependent changes, revealing a pronounced modality asymmetry. Hierarchical probing further shows that Camera-branch degradation extends from the visual backbone through the Camera-to-BEV pathway and is most severe at the pre-fusion Camera BEV representation. Guided by this diagnosis, we introduce unimodal-to-multimodal distillation at the degraded representation. This intervention consistently improves both Camera-branch Probe performance and full-model performance across all four tasks. These results show that multimodal gains can coexist with degraded unimodal representations, highlighting branch-level representation quality as an important dimension for evaluating and improving multimodal perception.
Robo-MUTUAL: Robotic Multimodal Task Specification via Unimodal Learning
Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院(AIR));College of Engineering, Peking University(北京大学工学院);Sensetime Research(商汤科技研究院);University of Toronto(多伦多大学);University College London(伦敦大学学院);Shanghai AI Lab(上海人工智能实验室)
ROVER: Benchmarking Reciprocal Cross-Modal Reasoning for Omnimodal Generation
University of Maryland, College Park(马里兰大学学院公园分校);University of Pennsylvania(宾夕法尼亚大学);The Hong Kong University of Science and Technology(香港科学与技术大学);University of Michigan(密歇根大学);University of Southern California(南加州大学)
MELINDA: A Multimodal Dataset for Biomedical Experiment Method Classification
University of California, Los Angeles(加利福尼亚大学洛杉矶分校);University of Southern California(南加利福尼亚大学);Intuit Inc.(英图伊特公司);Chan Zuckerberg Initiative(陈·扎克伯格倡议)
MSCR: Jointly Balancing Utilization Across Modalities and Discovering Synergistic Information
X-Humanoid;Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);Guangdong Laboratory of Artificial Intelligence and Digital Economy(广东省人工智能与数字经济实验室);University of Nottingham(诺丁汉大学);University of Electronic Science and Technology of China(电子科技大学)
Rethinking Vision Transformer and Masked Autoencoder in Multimodal Face Anti-Spoofing
Great Bay University(大湾区大学);Nanyang Technological University(南洋理工大学);University of Oulu(奥卢大学);Lappeenranta-Lahti University of Technology(拉彭兰塔-拉赫蒂理工大学);South China University of Technology(华南理工大学)
近年来,基于视觉Transformer(ViT)的多模态学习方法被提出来提高人脸反欺骗(FAS)系统的鲁棒性。然而,仍然没有工作来探索的基本性质(\textit{e.g.},模态感知输入,合适的多模态预训练,和有效的微调),在香草ViT多模态FAS。在本文中,我们研究了三个关键因素(即,输入、预训练和微调),用于具有RGB、红外(IR)和深度的多模态FAS。首先,在ViT输入方面,我们发现利用局部特征描述符有利于IR模态上的ViT,而不是RGB或深度模态。其次,在观察的效率直接微调整个或部分ViT,我们设计了一个自适应多模态适配器(AMA),它可以有效地聚合本地多模态功能,同时冻结大多数ViT参数。最后,考虑到任务(FAS与通用对象分类)和模态(多模态与单峰)的差距,ImageNet预训练模型对于多模态FAS任务可能是次优的。为了弥合这些差距,我们提出了模态非对称掩码自编码器(M $^{2}$ A $^{2}$ E),用于多模态FAS自监督预训练,而无需昂贵的注释标签。与先前的模态对称自动编码器相比,所提出的M $^{2}$ A $^{2}$ E能够学习更多内在的任务感知表示并且与模态不可知(例如,单峰、双峰和三峰)下游设置。在多模态FAS基准上进行的单峰(RGB,深度,IR)和多模态(RGB+深度,RGB+IR,深度+IR,RGB+深度+IR)设置的广泛实验证明了所提出的方法的优…
Aligning First, Then Fusing: A Novel Weakly Supervised Multimodal Violence Detection Method
School of Software Engineering(软件工程学院);Shenzhen Institute of Advanced Technology(深圳先进技术研究院);Xi’an Jiaotong University(西安交通大学);Chinese Academy of Sciences(中国科学院)
RoHyDR: Robust Hybrid Diffusion Recovery for Incomplete Multimodal Emotion Recognition
School of Computer Science and Engineering(计算机科学与工程学院);South China University of Technology(华南理工大学);School of Future Technology(未来技术学院);University of Oxford(牛津大学);Pengcheng Laboratory(鹏城实验室);Department of Computer Science(计算机科学系)
Diverse via bounded Agreement: Geometric Regularization for Multimodal Fusion
Department of Informatics University of Bern(伯尔尼大学信息学院);College of Computing and Data Science Nanyang Technological University(南洋理工大学计算机与数据科学学院);School of Software Engineering Xi’an Jiaotong University(西安交通大学软件工程学院)
Multimodal Inverse Attention Network with Intrinsic Discriminant Feature Exploitation for Fake News Detection
School of Information Science and Engineering, Shandong Normal University(山东师范大学信息科学与工程学院);Faculty of Engineering & Information Technology, University of Technology Sydney(悉尼科技大学工程与信息技术学院)
Unified Unsupervised Anomaly Detection via Matching Cost Filtering
State Key Laboratory of Synthetical Automation for Process Industries, Northeastern University, Shenyang, China(合成过程工业综合自动化国家重点实验室,东北大学,沈阳,中国);University of Surrey(Surrey大学);School of Computer Science, Wuhan University(武汉大学计算机学院);School of Computer Science, The University of Adelaide(阿德莱德大学计算机学院);College of Computing & Data Science, Nanyang Technological University(南洋理工大学计算机与数据科学学院);Surrey Institute for People-Centred Artificial Intelligence, and Centre for Vision, Speech and Signal Processing, University of Surrey(Surrey人本人工智能研究所,以及视觉、语音和信号处理中心,Surrey大学)