探索模态引导以增强基于VFM的特征融合用于3D语义分割中的无监督域适应
Exploring Modality Guidance to Enhance VFM-based Feature Fusion for UDA in 3D Semantic Segmentation
AI总结：
本研究利用视觉基础模型的跨域特征，通过图像与点云双流引导的融合网络，实现LiDAR 3D语义分割的无监督域适应，较最先进方法平均提升6.5 mIoU。
摘要：
视觉基础模型（VFMs）已成为许多下游视觉任务（如图像分类、图像分割和对象定位）的事实标准选择。然而，它们也能为可以利用跨模态信息（例如，来自配对图像数据）的下游3D任务提供显著效用。在我们的工作中，我们进一步探索了VFM在从标记源域适应到未标记目标域的任务（即基于LiDAR的3D语义分割）中的效用。我们的方法消耗配对的2D-3D（图像和点云）数据，并依赖来自VFM的稳健（跨域）特征，在标记源域和未标记目标域数据的混合上训练3D骨干网络。我们方法的核心是一个融合网络，该网络由图像和点云流共同引导，其相对贡献根据目标域进行调整。我们在多种设置下将我们提出的方法与不同的最先进方法进行了广泛比较，并取得了显著的性能提升。例如，与之前的最先进方法相比，平均提高了6.5 mIoU（在所有任务上）。
英文摘要：
Vision Foundation Models (VFMs) have become a de facto choice for many downstream vision tasks, like image classification, image segmentation, and object localization. However, they can also provide significant utility for downstream 3D tasks that can leverage the cross-modal information (e.g., from paired image data). In our work, we further explore the utility of VFMs for adapting from a labeled source to unlabeled target data for the task of LiDAR-based 3D semantic segmentation. Our method consumes paired 2D-3D (image and point cloud) data and relies on the robust (cross-domain) features from a VFM to train a 3D backbone on a mix of labeled source and unlabeled target data. At the heart of our method lies a fusion network that is guided by both the image and point cloud streams, with their relative contributions adjusted based on the target domain. We extensively compare our proposed methodology with different state-of-the-art methods in several settings and achieve strong performance gains. For example, achieving an average improvement of 6.5 mIoU (over all tasks), when compared with the previous state-of-the-art.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
