arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2303.12017cs.CV
收藏

基于双向相机-激光雷达融合的光流与场景流学习

Learning Optical Flow and Scene Flow with Bidirectional Camera-LiDAR Fusion

Nanjing University(南京大学)

Haisong Liu, Tao Lu, Yihui Xu, Jia Liu, Limin Wang

AI总结:

本文提出基于双向相机-激光雷达融合的端到端框架CamLiPWC和CamLiRAFT,联合估计光流与场景流,在多个基准上显著降低误差并排名领先。

摘要:

本文研究从同步的2D和3D数据中联合估计光流与场景流的问题。以往方法要么采用将联合任务拆分为独立阶段的复杂流程,要么以“早期融合”或“晚期融合”方式融合2D与3D信息。这种一刀切的方法面临两难困境:无法充分利用每种模态的特性,也无法最大化模态间的互补性。为解决该问题,我们提出一种新颖的端到端框架,该框架由2D和3D分支组成,并在特定层之间设置多个双向融合连接。与以往工作不同,我们采用基于点的3D分支提取激光雷达特征,因为它能保留点云的几何结构。为融合稠密图像特征与稀疏点特征,我们提出一种可学习的算子,称为双向相机-激光雷达融合模块(Bi-CLFM)。我们实例化了两种双向融合流程,一种基于金字塔式由粗到精架构(称为CamLiPWC),另一种基于循环全对场变换(称为CamLiRAFT)。在FlyingThings3D上,CamLiPWC和CamLiRAFT均超越了所有现有方法,并相比已发表的最佳结果实现了高达47.9%的3D端点误差降低。我们性能最佳的模型CamLiRAFT在KITTI Scene Flow基准上实现了4.26%的误差,在所有提交中排名第一,且参数量少得多。此外,我们的方法具有很强的泛化性能和处理非刚性运动的能力。代码可在https://github.com/MCG-NJU/CamLiFlow获取。

英文摘要:

In this paper, we study the problem of jointly estimating the optical flow and scene flow from synchronized 2D and 3D data. Previous methods either employ a complex pipeline that splits the joint task into independent stages, or fuse 2D and 3D information in an ``early-fusion'' or ``late-fusion'' manner. Such one-size-fits-all approaches suffer from a dilemma of failing to fully utilize the characteristic of each modality or to maximize the inter-modality complementarity. To address the problem, we propose a novel end-to-end framework, which consists of 2D and 3D branches with multiple bidirectional fusion connections between them in specific layers. Different from previous work, we apply a point-based 3D branch to extract the LiDAR features, as it preserves the geometric structure of point clouds. To fuse dense image features and sparse point features, we propose a learnable operator named bidirectional camera-LiDAR fusion module (Bi-CLFM). We instantiate two types of the bidirectional fusion pipeline, one based on the pyramidal coarse-to-fine architecture (dubbed CamLiPWC), and the other one based on the recurrent all-pairs field transforms (dubbed CamLiRAFT). On FlyingThings3D, both CamLiPWC and CamLiRAFT surpass all existing methods and achieve up to a 47.9\% reduction in 3D end-point-error from the best published result. Our best-performing model, CamLiRAFT, achieves an error of 4.26\% on the KITTI Scene Flow benchmark, ranking 1st among all submissions with much fewer parameters. Besides, our methods have strong generalization performance and the ability to handle non-rigid motion. Code is available at https://github.com/MCG-NJU/CamLiFlow.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑