发表机构
GFZ Helmholtz Center for Geosciences; INRAE, UMR TETIS, University of Montpellier; INRIA, EVERGREEN, University of Montpellier; CIRAD, UMR TETIS, University of Montpellier（德国波茨坦地学研究中心亥姆霍兹中心; 法国蒙彼利埃大学农业环境与生态研究院、蒙彼利埃大学TETIS联合研究单位; 法国蒙彼利埃大学信息与自动化研究所、EVERGREEN实验室; 法国蒙彼利埃大学国际农业研究磋商组织、蒙彼利埃大学TETIS联合研究单位）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对多模态分类中因操作约束导致训练和推理时模态可用性不一致、存在缺失任意模态的问题，提出多模态协同学习框架，引入特征和决策层面利用信息的两种方法，实验证明在不同缺失模态条件下有显著鲁棒性提升。
AI 中文摘要
多模态分类利用不同数据源的互补信息来提高预测性能。但现实场景中，如传感器故障或隐私限制等操作约束，会导致训练和推理时模态可用性不一致。以往研究主要针对双峰数据集且侧重于设计鲁棒融合过程。本文采用多模态协同学习框架，优先进行模态间协作而非融合。考虑到任何模态子集都可能缺失，不假设预定义的缺失模态模式（即缺失任意模态）。为此引入两种在特征和决策层面利用信息的方法。在两个多模态分类基准上的实验表明，在各种缺失模态条件下有显著的鲁棒性提升。第一种方法在单模态缺失的最小缺失条件下表现更鲁棒，第二种在除一个模态外所有模态都缺失的极端缺失条件下性能更好。
英文摘要
Multi-modal classification leverages complementary information across diverse data sources to enhance predictive performance. However, real-world scenarios subject to operational constraints, such as sensor failures or privacy restrictions, lead to inconsistent modality availability between training and inference times. To handle missing modalities, prior studies have mainly covered bimodal data setups and focused on designing robust fusion processes. Instead, we adopt a multi-modal co-learning framework that prioritizes inter-modal collaboration rather than multi-modal fusion. Specifically, we consider that any subset of modalities may be absent, without assuming predefined missing-modality patterns, an inference scenario we refer to as missing arbitrary modalities. To address this challenge, we introduce two alternative approaches that leverage information at both feature- and decision-level. Experiments on two multi-modal classification benchmarks demonstrate significant robustness gains in various missing modality conditions. The first method shows more robust behavior under minimal missing conditions, where a single modality is absent, whereas the second performs better under extreme missing conditions, where all-but-one modalities are missing. Our code is available at https://github.com/fmenat/Co4Miss.
CommentsAccepted at Discovery Science 2026
