CMDFusion：用于LIDAR语义分割的具有跨模态知识蒸馏的双向融合网络
CMDFusion: Bidirectional Fusion Network with Cross-modality Knowledge Distillation for LIDAR Semantic Segmentation
AI总结：
本文提出CMDFusion双向融合网络，通过2D到3D与3D到2D双向融合增强3D特征，并利用跨模态知识蒸馏使3D网络在推理时无需RGB图像即可生成2D信息，在SemanticKITTI和nuScenes上取得最佳性能。
摘要：
2D RGB图像和3D LIDAR点云为自动驾驶车辆的感知系统提供了互补知识。针对LIDAR语义分割任务，已有若干2D与3D融合方法被探索，但它们存在不同的问题。2D到3D融合方法在推理时需要严格配对的数据，而这在实际场景中可能无法获得；而3D到2D融合方法则无法显式地充分利用2D信息。因此，本文提出了一种具有跨模态知识蒸馏的双向融合网络（CMDFusion）。我们的方法有两个贡献。第一，我们的双向融合方案分别通过2D到3D融合和3D到2D融合显式地和隐式地增强3D特征，其性能优于任一单一融合方案。第二，我们将2D网络（相机分支）的2D知识蒸馏到3D网络（2D知识分支），使得3D网络即使对于不在相机FOV（视场）内的点也能生成2D信息。这样，由于2D知识分支根据3D LIDAR输入提供2D信息，推理时不再需要RGB图像。我们证明，在SemanticKITTI和nuScenes数据集上，我们的CMDFusion在所有基于融合的方法中取得了最佳性能。代码将在https://github.com/Jun-CEN/CMDFusion发布。
英文摘要：
2D RGB images and 3D LIDAR point clouds provide complementary knowledge for the perception system of autonomous vehicles. Several 2D and 3D fusion methods have been explored for the LIDAR semantic segmentation task, but they suffer from different problems. 2D-to-3D fusion methods require strictly paired data during inference, which may not be available in real-world scenarios, while 3D-to-2D fusion methods cannot explicitly make full use of the 2D information. Therefore, we propose a Bidirectional Fusion Network with Cross-Modality Knowledge Distillation (CMDFusion) in this work. Our method has two contributions. First, our bidirectional fusion scheme explicitly and implicitly enhances the 3D feature via 2D-to-3D fusion and 3D-to-2D fusion, respectively, which surpasses either one of the single fusion schemes. Second, we distillate the 2D knowledge from a 2D network (Camera branch) to a 3D network (2D knowledge branch) so that the 3D network can generate 2D information even for those points not in the FOV (field of view) of the camera. In this way, RGB images are not required during inference anymore since the 2D knowledge branch provides 2D information according to the 3D LIDAR input. We show that our CMDFusion achieves the best performance among all fusion-based methods on SemanticKITTI and nuScenes datasets. The code will be released at https://github.com/Jun-CEN/CMDFusion.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
