arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.10001cs.CV
收藏

视觉基础模型提升跨模态无监督域适应用于3D语义分割

Visual Foundation Models Boost Cross-Modal Unsupervised Domain Adaptation for 3D Semantic Segmentation

Fudan University(复旦大学);National University of Singapore(新加坡国立大学);Hochschule Bremerhaven(不来梅港应用科学大学)

Jingyi Xu, Weidong Yang, Lingdong Kong, Youquan Liu, Rui Zhang, Qingyuan Zhou, Ben Fei

AI总结:

提出VFM-Seg流程,利用2D视觉基础模型的先验知识生成更准确的伪标签,并通过跨模态融合和视锥体混合增强3D语义分割的无监督域适应性能。

摘要:

无监督域适应(UDA)对于减轻标注3D点云数据的工作量以及缓解面对新定义域时标签缺失的问题至关重要。近年来,出现了多种利用图像来增强跨域3D分割性能的方法。然而,由源域训练模型生成的伪标签,为未见域提供额外的监督信号,但由于其固有的噪声,在用于3D分割时并不充分,从而限制了神经网络的准确性。随着2D视觉基础模型(VFM)的出现及其丰富的先验知识,我们提出了一种新颖的流程VFM-Seg,通过利用这些模型来进一步增强跨模态无监督域适应框架。在这项工作中,我们研究如何利用VFM学到的知识先验,为未标注的目标域生成更准确的标签,并提高整体性能。我们首先利用一个在大规模图像-文本对上预训练的多模态VFM,为目标域的图像和点云提供监督标签(VFM-PL)。然后,采用另一个在细粒度2D掩码上训练的VFM来指导生成语义增强的图像和点云,以增强神经网络的性能,这些图像和点云混合了源域和目标域的数据,类似于视锥体(FrustumMixing)。最后,我们跨模态合并类级预测,为未标注的目标域生成更准确的标注。我们的方法在多个自动驾驶数据集上进行了评估,结果表明3D分割任务取得了显著改进。

英文摘要:

Unsupervised domain adaptation (UDA) is vital for alleviating the workload of labeling 3D point cloud data and mitigating the absence of labels when facing a newly defined domain. Various methods of utilizing images to enhance the performance of cross-domain 3D segmentation have recently emerged. However, the pseudo labels, which are generated from models trained on the source domain and provide additional supervised signals for the unseen domain, are inadequate when utilized for 3D segmentation due to their inherent noisiness and consequently restrict the accuracy of neural networks. With the advent of 2D visual foundation models (VFMs) and their abundant knowledge prior, we propose a novel pipeline VFMSeg to further enhance the cross-modal unsupervised domain adaptation framework by leveraging these models. In this work, we study how to harness the knowledge priors learned by VFMs to produce more accurate labels for unlabeled target domains and improve overall performance. We first utilize a multi-modal VFM, which is pre-trained on large scale image-text pairs, to provide supervised labels (VFM-PL) for images and point clouds from the target domain. Then, another VFM trained on fine-grained 2D masks is adopted to guide the generation of semantically augmented images and point clouds to enhance the performance of neural networks, which mix the data from source and target domains like view frustums (FrustumMixing). Finally, we merge class-wise prediction across modalities to produce more accurate annotations for unlabeled target domains. Our method is evaluated on various autonomous driving datasets and the results demonstrate a significant improvement for 3D segmentation task.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑