时间会证明一切:时间多视图三维目标检测的新展望和基线
Time Will Tell: New Outlooks and A Baseline for Temporal Multi-View 3D Object Detection
摘要:
虽然最近的仅摄像机3D检测方法利用多个时间步长,但它们使用的有限历史显著阻碍了时间融合可以改善对象感知的程度。观察到现有的作品的多帧图像的融合是时间立体匹配的实例,我们发现性能受到1)匹配分辨率的低粒度和2)由有限的历史使用产生的次优多视图设置之间的相互作用的阻碍。我们的理论和实证分析表明,视图之间的最佳时间差对于不同的像素和深度变化很大,因此有必要在长期历史中融合许多时间步长。基于我们的调查,我们建议从长期的图像观测历史中生成成本卷,用更优化的多视图匹配设置补偿粗糙但有效的匹配分辨率。此外,我们增加了用于长期,粗匹配与短期,细粒度匹配的每帧单目深度预测,并发现长期和短期的时间融合是高度互补的。在保持高效率的同时,我们的框架在nuScenes上设置了新的最先进的技术,在测试集上获得了第一名,并在验证集上以5.2%的mAP和3.7%的NDS超过了之前的最佳技术。代码将发布$\href{https://github.com/Divadi/SOLOFusion}{here.}$
英文摘要:
While recent camera-only 3D detection methods leverage multiple timesteps, the limited history they use significantly hampers the extent to which temporal fusion can improve object perception. Observing that existing works' fusion of multi-frame images are instances of temporal stereo matching, we find that performance is hindered by the interplay between 1) the low granularity of matching resolution and 2) the sub-optimal multi-view setup produced by limited history usage. Our theoretical and empirical analysis demonstrates that the optimal temporal difference between views varies significantly for different pixels and depths, making it necessary to fuse many timesteps over long-term history. Building on our investigation, we propose to generate a cost volume from a long history of image observations, compensating for the coarse but efficient matching resolution with a more optimal multi-view matching setup. Further, we augment the per-frame monocular depth predictions used for long-term, coarse matching with short-term, fine-grained matching and find that long and short term temporal fusion are highly complementary. While maintaining high efficiency, our framework sets new state-of-the-art on nuScenes, achieving first place on the test set and outperforming previous best art by 5.2% mAP and 3.7% NDS on the validation set. Code will be released $\href{https://github.com/Divadi/SOLOFusion}{here.}$
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。