arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2111.08313cs.CV
收藏

迈向全面的单目深度估计:多头优于单头

Towards Comprehensive Monocular Depth Estimation: Multiple Heads Are Better Than One

Beihang University(北京航空航天大学);Amazon(亚马逊)

Shuwei Shao, Ran Li, Zhongcai Pei, Zhong Liu, Weihai Chen, Wentao Zhu, Xingming Wu, Baochang Zhang

AI总结:

本文提出融合多个基于 Transformer 与 CNN 的弱深度预测器,并通过混合器自适应集成,构建 TEDepth,在 NYU-Depth-v2、KITTI 及跨数据集测试中取得先进性能。

摘要:

深度估计在计算机视觉领域引起了广泛关注。然而,仅使用一张 RGB 图像恢复准确的深度图仍然相当困难。我们观察到一种现象:由于网络架构、损失函数等方面的差异,现有方法往往会在不同情况下失败。在本工作中,我们对这一现象进行了研究,并提出整合多个弱深度预测器的优势,以构建一个全面且准确的深度预测器,这对于许多现实世界应用(例如三维重建)至关重要。具体而言,我们利用不同的基于 Transformer 和基于卷积神经网络(CNN)的架构,构建了多个基础(弱)深度预测器。Transformer 建立长程相关性,而 CNN 则保留因空间归纳偏置而被 Transformer 忽略的局部信息。因此,Transformer 与 CNN 的耦合有助于生成互补的深度估计,这对于实现全面的深度预测器至关重要。随后,我们设计了混合器,以从多个弱预测中学习,并将其自适应地融合为一个强深度估计。所得模型被我们称为 Transformer-assisted depth ensembles(TEDepth)。在标准的 NYU-Depth-v2 和 KITTI 数据集上,我们全面探究了神经集成如何影响深度估计,并证明了 TEDepth 取得了优于以往最先进方法的结果。为了验证跨相机的泛化能力,我们将在 NYU-Depth-v2 上训练的模型直接应用于 SUN RGB-D 数据集,而未进行任何微调;优越的结果突显了其强大的泛化能力。

英文摘要:

Depth estimation attracts widespread attention in the computer vision community. However, it is still quite difficult to recover an accurate depth map using only one RGB image. We observe a phenomenon that existing methods tend to fail in different cases, caused by differences in network architecture, loss function and so on. In this work, we investigate into the phenomenon and propose to integrate the strengths of multiple weak depth predictor to build a comprehensive and accurate depth predictor, which is critical for many real-world applications, e.g., 3D reconstruction. Specifically, we construct multiple base (weak) depth predictors by utilizing different Transformer-based and convolutional neural network (CNN)-based architectures. Transformer establishes long-range correlation while CNN preserves local information ignored by Transformer due to the spatial inductive bias. Therefore, the coupling of Transformer and CNN contributes to the generation of complementary depth estimates, which are essential to achieve a comprehensive depth predictor. Then, we design mixers to learn from multiple weak predictions and adaptively fuse them into a strong depth estimate. The resultant model, which we refer to as Transformer-assisted depth ensembles (TEDepth). On the standard NYU-Depth-v2 and KITTI datasets, we thoroughly explore how the neural ensembles affect the depth estimation and demonstrate that our TEDepth achieves better results than previous state-of-the-art approaches. To validate the generalizability across cameras, we directly apply the models trained on NYU-Depth-v2 to the SUN RGB-D dataset without any fine-tuning, and the superior results emphasize its strong generalizability.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑