通过单模态与跨模态蒸馏学习鲁棒的任模态分割器
Learning Robust Anymodal Segmentor with Unimodal and Cross-modal Distillation
AI总结：
针对多模态分割器的单模态偏差问题，提出含单模态、跨模态及预测层语义蒸馏的任模态鲁棒分割框架，可处理任意视觉模态组合，在多类基准上取得优异性能。
摘要：
同时利用多传感器的多模态输入训练分割器在直觉上具有优势，但实际应用中颇具挑战。核心挑战之一是单模态偏差，即多模态分割器过度依赖某些模态，当其他模态缺失时性能会下降，这种情况在现实应用中十分常见。为此，我们提出了首个用于学习鲁棒分割器的框架，该框架能够处理视觉模态的任意组合。具体而言，我们首先引入一种并行多模态学习策略来训练一个强教师模型。随后，通过在多尺度表征空间中将多模态分割器的特征层知识迁移到任模态分割器，实现跨模态与单模态蒸馏，旨在解决单模态偏差问题，避免对特定模态的过度依赖。此外，我们还提出了一种预测层的模态无关语义蒸馏方法，以实现分割任务的语义知识迁移。在合成与真实世界多传感器基准上的大量实验表明，我们的方法取得了优异的性能。
英文摘要：
Simultaneously using multimodal inputs from multiple sensors to train segmentors is intuitively advantageous but practically challenging. A key challenge is unimodal bias, where multimodal segmentors over rely on certain modalities, causing performance drops when others are missing, common in real world applications. To this end, we develop the first framework for learning robust segmentor that can handle any combinations of visual modalities. Specifically, we first introduce a parallel multimodal learning strategy for learning a strong teacher. The cross-modal and unimodal distillation is then achieved in the multi scale representation space by transferring the feature level knowledge from multimodal to anymodal segmentors, aiming at addressing the unimodal bias and avoiding over-reliance on specific modalities. Moreover, a prediction level modality agnostic semantic distillation is proposed to achieve semantic knowledge transferring for segmentation. Extensive experiments on both synthetic and real-world multi-sensor benchmarks demonstrate that our method achieves superior performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
