CMR-Agent：学习用于迭代式图像到点云配准的跨模态智能体
CMR-Agent: Learning a Cross-Modal Agent for Iterative Image-to-Point Cloud Registration
AI总结：
本文将图像到点云配准建模为迭代马尔可夫决策过程，提出结合强化学习与模仿学习的跨模态智能体CMR-Agent，通过2D-3D混合状态表示和一次性嵌入复用，在KITTI-Odometry与NuScenes上实现了高精度且高效的配准。
摘要：
图像到点云配准旨在确定RGB图像相对于点云的相机位姿，在预建LiDAR地图中的相机定位任务中发挥着重要作用。尽管存在模态差异，大多数基于学习的方法在特征空间中建立2D-3D点对应关系，却缺乏用于迭代优化的反馈机制，导致精度和可解释性较差。本文提出将配准过程重新表述为迭代马尔可夫决策过程，允许基于每个中间状态对相机位姿进行增量调整。为此，我们采用强化学习开发了一个跨模态配准智能体（CMR-Agent），并使用模仿学习初始化其配准策略，以保证训练的稳定性和快速启动。针对跨模态观测，我们提出了一种2D-3D混合状态表示，充分利用RGB图像的细粒度特征，同时减少由相机视锥空间截断引起的无用中性状态。此外，整体框架经过精心设计，能够高效复用一次性跨模态嵌入，避免重复且耗时的特征提取。在KITTI-Odometry和NuScenes数据集上的大量实验表明，CMR-Agent在配准精度和效率上均具有竞争力。一旦完成一次性嵌入，每次迭代仅需几毫秒。
英文摘要：
Image-to-point cloud registration aims to determine the relative camera pose of an RGB image with respect to a point cloud. It plays an important role in camera localization within pre-built LiDAR maps. Despite the modality gaps, most learning-based methods establish 2D-3D point correspondences in feature space without any feedback mechanism for iterative optimization, resulting in poor accuracy and interpretability. In this paper, we propose to reformulate the registration procedure as an iterative Markov decision process, allowing for incremental adjustments to the camera pose based on each intermediate state. To achieve this, we employ reinforcement learning to develop a cross-modal registration agent (CMR-Agent), and use imitation learning to initialize its registration policy for stability and quick-start of the training. According to the cross-modal observations, we propose a 2D-3D hybrid state representation that fully exploits the fine-grained features of RGB images while reducing the useless neutral states caused by the spatial truncation of camera frustum. Additionally, the overall framework is well-designed to efficiently reuse one-shot cross-modal embeddings, avoiding repetitive and time-consuming feature extraction. Extensive experiments on the KITTI-Odometry and NuScenes datasets demonstrate that CMR-Agent achieves competitive accuracy and efficiency in registration. Once the one-shot embeddings are completed, each iteration only takes a few milliseconds.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
