MMANet：面向不完整多模态学习的边缘感知蒸馏与模态感知正则化
MMANet: Margin-aware Distillation and Modality-aware Regularization for Incomplete Multimodal Learning
AI总结：
本文提出MMANet框架，通过边缘感知蒸馏和模态感知正则化解决不完整多模态学习中模态缺失导致的性能下降问题，并在分类与分割任务上取得显著提升。
摘要：
多模态学习在众多场景中展现出巨大潜力，近年来吸引了越来越多的关注。然而，它经常面临模态数据缺失的问题，从而在实际应用中遭受严重的性能下降。为此，我们提出了一个名为MMANet的通用框架，以辅助不完整多模态学习。它由三个组件组成：用于推理的部署网络、将全面的多模态信息传递给部署网络的教师网络，以及引导部署网络平衡弱模态组合的正则化网络。具体而言，我们提出了一种新颖的边缘感知蒸馏（MAD）方法，通过用分类不确定性加权样本贡献来辅助信息传递。这促使部署网络关注决策边界附近的样本，并获得精细的类间边缘。此外，我们设计了一种模态感知正则化（MAR）算法来挖掘弱模态组合，并引导正则化网络为它们计算预测损失。这迫使部署网络自适应地提升其对弱模态组合的表示能力。最后，在多模态分类和分割任务上的大量实验表明，我们的MMANet显著优于最先进的方法。代码可在以下地址获取：https://github.com/shicaiwei123/MMANet
英文摘要：
Multimodal learning has shown great potentials in numerous scenes and attracts increasing interest recently. However, it often encounters the problem of missing modality data and thus suffers severe performance degradation in practice. To this end, we propose a general framework called MMANet to assist incomplete multimodal learning. It consists of three components: the deployment network used for inference, the teacher network transferring comprehensive multimodal information to the deployment network, and the regularization network guiding the deployment network to balance weak modality combinations. Specifically, we propose a novel margin-aware distillation (MAD) to assist the information transfer by weighing the sample contribution with the classification uncertainty. This encourages the deployment network to focus on the samples near decision boundaries and acquire the refined inter-class margin. Besides, we design a modality-aware regularization (MAR) algorithm to mine the weak modality combinations and guide the regularization network to calculate prediction loss for them. This forces the deployment network to improve its representation ability for the weak modality combinations adaptively. Finally, extensive experiments on multimodal classification and segmentation tasks demonstrate that our MMANet outperforms the state-of-the-art significantly. Code is available at: https://github.com/shicaiwei123/MMANet
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
