残差引导的专家特化用于不完整多模态学习
Residual-Guided Expert Specialization for Incomplete Multimodal Learning
AI总结：
提出MARS框架，利用缺失引起的表示偏差引导专家特化，通过残差信号路由样本到对应专家，并引入差异感知噪声正则化缓解训练-测试路由差距，在缺失场景下提升多模态分类与分割性能。
摘要：
由于现实世界中的预测系统在推理时经常面临模态缺失，不完整多模态学习（IML）仍然是一个实际挑战。尽管先前的方法旨在学习对缺失输入鲁棒的表示，但由于缺失证据，来自不完整模态的表示不可避免地偏离其完整模态对应物。为了显式利用这些偏差，我们提出了MARS（缺失感知残差引导特化），一种专家混合框架，根据缺失如何重塑表示来引导专家特化。在训练过程中，通过将来自不完整输入的任务表示与其完整对应物进行对比，我们推导出一个特权残差信号，该信号捕获了这种表示差距。残差信号引导一个残差路由器将样本分配给针对相应偏差模式特化的专家。同时，一个特征路由器学习仅使用不完整输入模仿这种路由行为，从而无需访问完整模态即可部署。为了缓解这种训练-测试路由器差距，我们开发了一种差异感知噪声正则化，当特征路由器偏离时，自适应地扰动残差路由器的决策，从而在不完美模仿下增强专家鲁棒性。在缺失场景下的多模态分类（CASIA-SURF、CREMA-D、UPMC Food-101）和分割（MCubeS）实验表明，MARS持续超越基线，同时保持高效性并可扩展到多种骨干网络和任务。
英文摘要：
As real-world prediction systems often face missing modalities at inference, incomplete multimodal learning (IML) remains a practical challenge. While prior methods aim to learn representations robust to missing inputs, representations from incomplete modalities inevitably deviate from their full-modality counterparts due to missing evidence. To explicitly leverage these deviations, we propose MARS (Missingness-Aware Residual-guided Specialization), a mixture-of-experts framework that guides expert specialization based on how representations are reshaped by missingness. By contrasting task representations derived from incomplete inputs with their complete counterparts during training, we derive a privileged residual signal that captures this representational gap. The residual signal guides a residual router to assign samples to experts specialized for the corresponding deviation patterns. In parallel, a feature router learns to imitate this routing behavior using only incomplete inputs, enabling deployment without access to full modalities. To mitigate this train-test router gap, we develop a discrepancy-aware noise regularization that adaptively perturbs the residual router's decisions when the feature router deviates, enhancing expert robustness under imperfect imitation. Experiments on multimodal classification (CASIA-SURF, CREMA-D, UPMC Food-101) and segmentation (MCubeS) under missing scenarios show that MARS consistently surpasses baselines while remaining efficient and extensible to diverse backbones and tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
