arXiv 2310.034202024-04-13cs.CV

FreeReg：利用预训练扩散模型和单目深度估计器的图像到点云配准

FreeReg: Image-to-Point Cloud Registration Leveraging Pretrained Diffusion Models and Monocular Depth Estimators

LISMARS；Hubei Luojia Laboratory（湖北珞珈实验室）；Wuhan University（武汉大学）；The University of Hong Kong（香港大学）；The Hong Kong Polytechnic University（香港理工大学）；Texas A&M University（德克萨斯农工大学）

Haiping Wang, Yuan Liu, Bing Wang, Yujing Sun, Zhen Dong, Wenping Wang, Bisheng Yang

备注：: CameraReady version for ICLR 2024. Project Page: https://whu-usi3dv.github.io/FreeReg/

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对图像与点云跨模态特征匹配难的问题，提出FreeReg方法，借助预训练扩散模型的扩散特征和单目深度估计的几何特征实现无任务特定训练的精准配准，在多个基准上性能显著优于现有最优方法。

摘要：

图像与点云之间的跨模态特征匹配是图像到点云配准的核心问题。然而，由于图像和点云存在模态差异，现有用于特征匹配的度量学习方法难以学习到鲁棒且具判别性的跨模态特征。本文并未在跨模态数据上应用度量学习，而是提出先通过预训练大规模模型统一图像与点云的模态，再在同模态下建立鲁棒的对应关系。研究表明，由深度到图像扩散模型提取的中间特征（称为扩散特征）在图像和点云间具有语义一致性，可构建粗略但鲁棒的跨模态对应关系。本文进一步在单目深度估计器生成的深度图上提取几何特征，通过匹配这类几何特征，显著提升了扩散特征生成的粗略对应关系的精度。大量实验证明，无需任何任务特定训练，直接结合两种特征即可实现精准的图像到点云配准。在三个公开的室内外基准测试集上，所提方法的内点率平均提升20.6%，内点数量达到现有最优方法的三倍，配准召回率平均提升48.6%。

英文摘要：

Matching cross-modality features between images and point clouds is a fundamental problem for image-to-point cloud registration. However, due to the modality difference between images and points, it is difficult to learn robust and discriminative cross-modality features by existing metric learning methods for feature matching. Instead of applying metric learning on cross-modality data, we propose to unify the modality between images and point clouds by pretrained large-scale models first, and then establish robust correspondence within the same modality. We show that the intermediate features, called diffusion features, extracted by depth-to-image diffusion models are semantically consistent between images and point clouds, which enables the building of coarse but robust cross-modality correspondences. We further extract geometric features on depth maps produced by the monocular depth estimator. By matching such geometric features, we significantly improve the accuracy of the coarse correspondences produced by diffusion features. Extensive experiments demonstrate that without any task-specific training, direct utilization of both features produces accurate image-to-point cloud registration. On three public indoor and outdoor benchmarks, the proposed method averagely achieves a 20.6 percent improvement in Inlier Ratio, a three-fold higher Inlier Number, and a 48.6 percent improvement in Registration Recall than existing state-of-the-arts.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。