Introducing a Class-Aware Metric for Monocular Depth Estimation: An Automotive Perspective
Dr. Ing. h.c. F. Porsche AG（保时捷股份公司）；Ulm University（乌尔姆大学）；Porsche Engineering Group GmbH（保时捷工程集团有限公司）；Stuttgart Media University（斯图加特媒体大学）；University of Freiburg（弗莱堡大学）
Tim Bader, Leon Eisemann, Adrian Pogorzelski, Namrata Jangid, Attila-Balazs Kis
备注：
Accepted at the European Conference on Computer Vision (ECCV) 2024 Workshop on Out Of Distribution Generalization in Computer Vision
The increasing accuracy reports of metric monocular depth estimation models lead to a growing interest from the automotive domain. Current model evaluations do not provide deeper insights into the models' performance, also in relation to safety-critical or unseen classes. Within this paper, we present a novel approach for the evaluation of depth estimation models. Our proposed metric leverages three components, a class-wise component, an edge and corner image feature component, and a global consistency retaining component. Classes are further weighted on their distance in the scene and on criticality for automotive applications. In the evaluation, we present the benefits of our metric through comparison to classical metrics, class-wise analytics, and the retrieval of critical situations. The results show that our metric provides deeper insights into model results while fulfilling safety-critical requirements. We release the code and weights on the following repository: https://github.com/leisemann/ca_mmde
German Research Center for Artificial Intelligence – DFKI（德国人工智能研究中心–DFKI）；BMW Group（宝马集团）
Contrary to the ongoing trend in automotive applications towards usage of more diverse and more sensors, this work tries to solve the complex scene flow problem under a monocular camera setup, i.e. using a single sensor. Towards this end, we exploit the latest achievements in single image depth estimation, optical flow, and sparse-to-dense interpolation and propose a monocular combination approach (MonoComb) to compute dense scene flow. MonoComb uses optical flow to relate reconstructed 3D positions over time and interpolates occluded areas. This way, existing monocular methods are outperforme…
The Allstate Corporation（好事达公司）；Stanford University（斯坦福大学）
Many automotive applications, such as Advanced Driver Assistance Systems (ADAS) for collision avoidance and warnings, require estimating the future automotive risk of a driving scene. We present a low-cost system that predicts the collision risk over an intermediate time horizon from a monocular video source, such as a dashboard-mounted camera. The modular system includes components for object detection, object tracking, and state estimation. We introduce solutions to the object tracking and distance estimation problems. Advanced approaches to the other tasks are used to produce real-time pred…
Vision-based depth estimation is a key feature in autonomous systems, which often relies on a single camera or several independent ones. In such a monocular setup, dense depth is obtained with either additional input from one or several expensive LiDARs, e.g., with 64 beams, or camera-only methods, which suffer from scale-ambiguity and infinite-depth problems. In this paper, we propose a new alternative of densely estimating metric depth by combining a monocular camera with a light-weight LiDAR, e.g., with 4 beams, typical of today's automotive-grade mass-produced laser scanners. Inspired by r…
MoD-SLAM: Monocular Dense Mapping for Unbounded 3D Scene Reconstruction
Columbia University（哥伦比亚大学）；Cloudspace Technology（云空间科技）；The University of Tokyo（东京大学）；Dianjing Ciyuan Culture（电境慈源文化）；Dalian University of Technology（大连理工大学）
Zhejiang University（浙江大学）；Horizon Robotics（地平线机器人）；The University of Technology, Sydney（悉尼科技大学）
Many standard robotic platforms are equipped with at least a fixed 2D laser range finder and a monocular camera. Although those platforms do not have sensors for 3D depth sensing capability, knowledge of depth is an essential part in many robotics activities. Therefore, recently, there is an increasing interest in depth estimation using monocular images. As this task is inherently ambiguous, the data-driven estimated depth might be unreliable in robotics applications. In this paper, we have attempted to improve the precision of monocular depth estimation by introducing 2D planar observation fr…
Recent automotive vision work has focused almost exclusively on processing forward-facing cameras. However, future autonomous vehicles will not be viable without a more comprehensive surround sensing, akin to a human driver, as can be provided by 360° panoramic cameras. We present an approach to adapt contemporary deep network architectures developed on conventional rectilinear imagery to work on equirectangular 360° panoramic imagery. To address the lack of annotated panoramic automotive datasets availability, we adapt a contemporary automotive dataset, via style and projection transformation…
Pose estimation is a fundamental building block for robotic applications such as autonomous vehicles, UAV, and large scale augmented reality. It is also a prohibitive factor for those applications to be in mass production, since the state-of-the-art, centimeter-level pose estimation often requires long mapping procedures and expensive localization sensors, e.g. LiDAR and high precision GPS/IMU, etc. To overcome the cost barrier, we propose a neural network based solution to localize a consumer degree RGB camera within a prior sparse LiDAR map with comparable centimeter-level precision. We achi…
We present animatable neural radiance fields (animatable NeRF) for detailed human avatar creation from monocular videos. Our approach extends neural radiance fields (NeRF) to the dynamic scenes with human movements via introducing explicit pose-guided deformation while learning the scene representation network. In particular, we estimate the human pose for each frame and learn a constant canonical space for the detailed human template, which enables natural shape deformation from the observation space to the canonical space under the explicit control of the pose parameters. To compensate for i…
School of Computing Sciences（计算科学学院）；University of East Anglia（东英吉利大学）
We propose SUB-Depth, a universal multi-task training framework for self-supervised monocular depth estimation (SDE). Depth models trained with SUB-Depth outperform the same models trained in a standard single-task SDE framework. By introducing an additional self-distillation task into a standard SDE training framework, SUB-Depth trains a depth network, not only to predict the depth map for an image reconstruction task, but also to distill knowledge from a trained teacher network with unlabelled data. To take advantage of this multi-task setting, we propose homoscedastic uncertainty formulatio…
University of Oulu（奥卢大学）；Czech Technical University in Prague（布拉格捷克理工大学）；Tampere University（坦佩雷大学）
In this paper, we propose enhancing monocular depth estimation by adding 3D points as depth guidance. Unlike existing depth completion methods, our approach performs well on extremely sparse and unevenly distributed point clouds, which makes it agnostic to the source of the 3D points. We achieve this by introducing a novel multi-scale 3D point fusion network that is both lightweight and efficient. We demonstrate its versatility on two different depth estimation problems where the 3D points have been acquired with conventional structure-from-motion and LiDAR. In both cases, our network performs…
Qing Yuan Research Institute（庆元研究院）；MoE Key Lab of Artificial Intelligence（教育部人工智能重点实验室）；AI Institute（人工智能研究院）；Shanghai Jiao Tong University（上海交通大学）
General object grasping is an important yet unsolved problem in the field of robotics. Most of the current methods either generate grasp poses with few DoF that fail to cover most of the success grasps, or only take the unstable depth image or point cloud as input which may lead to poor results in some cases. In this paper, we propose RGBD-Grasp, a pipeline that solves this problem by decoupling 7-DoF grasp detection into two sub-tasks where RGB and depth information are processed separately. In the first stage, an encoder-decoder like convolutional neural network Angle-View Net(AVN) is propos…
Nanjing University of Science and Technology（南京理工大学）；Renmin University of China（中国人民大学）；Nreal（Nreal（太若科技））
Monocular 3D object detection is a fundamental but very important task to many applications including autonomous driving, robotic grasping and augmented reality. Existing leading methods tend to estimate the depth of the input image first, and detect the 3D object based on point cloud. This routine suffers from the inherent gap between depth estimation and object detection. Besides, the prediction error accumulation would also affect the performance. In this paper, a novel method named MonoSIM is proposed. The insight behind introducing MonoSIM is that we propose to simulate the feature learni…
The Computer Vision Center (CVC)（计算机视觉中心）；Universitat Autònoma de Barcelona (UAB)（巴塞罗那自治大学）；Huawei European Research Center（华为欧洲研究中心）
Depth estimation provides essential information to perform autonomous driving and driver assistance. Especially, Monocular Depth Estimation is interesting from a practical point of view, since using a single camera is cheaper than many other options and avoids the need for continuous calibration strategies as required by stereo-vision approaches. State-of-the-art methods for Monocular Depth Estimation are based on Convolutional Neural Networks (CNNs). A promising line of work consists of introducing additional semantic information about the traffic scene when training CNNs for depth estimation…
Beijing University of Posts and Telecommunications（北京邮电大学）
It is an exciting task to recover the scene's 3d-structure and camera pose from the video sequence. Most of the current solutions divide it into two parts, monocular depth recovery and camera pose estimation. The monocular depth recovery is often studied as an independent part, and a better depth estimation is used to solve the pose. While camera pose is still estimated by traditional SLAM (Simultaneous Localization And Mapping) methods in most cases. The use of unsupervised method for monocular depth recovery and pose estimation has benefited from the study of [1] and achieved good results. I…
We introduce SharpNet, a method that predicts an accurate depth map for an input color image, with a particular attention to the reconstruction of occluding contours: Occluding contours are an important cue for object recognition, and for realistic integration of virtual objects in Augmented Reality, but they are also notoriously difficult to reconstruct accurately. For example, they are a challenge for stereo-based reconstruction methods, as points around an occluding contour are visible in only one image. Inspired by recent methods that introduce normal estimation to improve depth prediction…
Peking University（北京大学）；Advanced Institute of Information Technology, Peking University（北京大学信息工程高等研究院）；Information Technology R&D Innovation Center of Peking University（北京大学信息技术研发创新中心）
We present two versatile methods to generally enhance self-supervised monocular depth estimation (MDE) models. The high generalizability of our methods is achieved by solving the fundamental and ubiquitous problems in photometric loss function. In particular, from the perspective of spatial frequency, we first propose Ambiguity-Masking to suppress the incorrect supervision under photometric loss at specific object boundaries, the cause of which could be traced to pixel-level ambiguity. Second, we present a novel frequency-adaptive Gaussian low-pass filter, designed to robustify the photometric…
Peking University（北京大学）；Advanced Institute of Information Technology, Peking University（北京大学先进信息技术研究院）；Information Technology R&D Innovation Center of Peking University（北京大学信息技术研发创新中心）
Self-supervised monocular depth estimation (MDE) models universally suffer from the notorious edge-fattening issue. Triplet loss, as a widespread metric learning strategy, has largely succeeded in many computer vision applications. In this paper, we redesign the patch-based triplet loss in MDE to alleviate the ubiquitous edge-fattening issue. We show two drawbacks of the raw triplet loss in MDE and demonstrate our problem-driven redesigns. First, we present a min. operator based strategy applied to all negative samples, to prevent well-performing negatives sheltering the error of edge-fattenin…
Beijing Jiaotong University（北京交通大学）；UISEE Technology (Beijing) Co., Ltd.（UISEE科技（北京）有限公司）；City University of Hong Kong（香港城市大学）
Depth map super-resolution is a task with high practical application requirements in the industry. Existing color-guided depth map super-resolution methods usually necessitate an extra branch to extract high-frequency detail information from RGB image to guide the low-resolution depth map reconstruction. However, because there are still some differences between the two modalities, direct information transmission in the feature dimension or edge map dimension cannot achieve satisfactory result, and may even trigger texture copying in areas where the structures of the RGB-D pair are inconsistent…
Bidirectional Regression for Monocular 6DoF Head Pose Estimation and Reference System Alignment
Department of Electronics and Communications Engineering, Kwangwoon University（韩国 Kwangwoon 大学电子与通信工程系）；Department of Software Engineering, Dankook University（韩国 Dankook 大学软件工程系）；School of Computer Science, University of Birmingham（英国 Birmingham 大学计算机科学学院）
This work introduces an evaluation benchmark for depth estimation and completion using high-resolution depth measurements with angular resolution of up to 25" (arcsecond), akin to a 50 megapixel camera with per-pixel depth available. Existing datasets, such as the KITTI benchmark, provide only sparse reference measurements with an order of magnitude lower angular resolution - these sparse measurements are treated as ground truth by existing depth estimation methods. We propose an evaluation methodology in four characteristic automotive scenarios recorded in varying weather conditions (day, nig…
Gated cameras hold promise as an alternative to scanning LiDAR sensors with high-resolution 3D depth that is robust to back-scatter in fog, snow, and rain. Instead of sequentially scanning a scene and directly recording depth via the photon time-of-flight, as in pulsed LiDAR sensors, gated imagers encode depth in the relative intensity of a handful of gated slices, captured at megapixel resolution. Although existing methods have shown that it is possible to decode high-resolution depth from such measurements, these methods require synchronized and calibrated LiDAR to supervise the gated depth …
Hong Kong University of Science and Technology（香港科技大学）；Tencent Inc（腾讯公司）
A monocular 3D object tracking system generally has only up-to-scale pose estimation results without any prior knowledge of the tracked object. In this paper, we propose a novel idea to recover the metric scale of an arbitrary dynamic object by optimizing the trajectory of the objects in the world frame, without motion assumptions. By introducing an additional constraint in the time domain, our monocular visual-inertial tracking system can obtain continuous six degree of freedom (6-DoF) pose estimation without scale ambiguity. Our method requires neither fixed multi-camera nor depth sensor set…
Valeo DAR Kronach（法雷奥达尔斯克罗纳赫）；ENSTA ParisTech Palaiseau（巴黎高等理工学院帕莱索校区）；Valeo Vision Systems（法雷奥视觉系统公司）；Technische Universität Ilmenau（伊尔梅瑙工业大学）
Fisheye cameras are commonly used in applications like autonomous driving and surveillance to provide a large field of view ($>180^{\circ}$). However, they come at the cost of strong non-linear distortions which require more complex algorithms. In this paper, we explore Euclidean distance estimation on fisheye cameras for automotive scenes. Obtaining accurate and dense depth supervision is difficult in practice, but self-supervised learning approaches show promising results and could potentially overcome the problem. We present a novel self-supervised scale-aware framework for learning Euclide…
Physics-Grounded Monocular Vehicle Distance Estimation Using Standardized License Plate Typography
Department of Computer and Information Science, University of Michigan-Dearborn（1计算机与信息科学系，密歇根大学-迪尔伯恩分校）；Department of Industrial and Manufacturing Systems Engineering, University of Michigan-Dearborn（2工业与制造系统工程系，密歇根大学-迪尔伯恩分校）
Mapillary Research（Mapillary研究）；University of Trento（特伦托大学）；Fondazione Bruno Kessler（布鲁诺·凯斯勒基金会）
While expensive LiDAR and stereo camera rigs have enabled the development of successful 3D object detection methods, monocular RGB-only approaches lag much behind. This work advances the state of the art by introducing MoVi-3D, a novel, single-stage deep architecture for monocular 3D object detection. MoVi-3D builds upon a novel approach which leverages geometrical information to generate, both at training and test time, virtual views where the object appearance is normalized with respect to distance. These virtually generated views facilitate the detection task as they significantly reduce th…
Towards Robust Driving Perception: A Flexible Scale-Driven Family for Self-Supervised Monocular Depth Estimation
Hefei University of Technology（合肥工业大学）；Nanjing University of Posts and Telecommunications（南京邮电大学）；Chengpin Home Tech（橙品家居科技）；Differential Robotics（差分机器人）
Tongji University（同济大学）；National University of Singapore（新加坡国立大学）
In recent years, self-supervised methods for monocular depth estimation has rapidly become an significant branch of depth estimation task, especially for autonomous driving applications. Despite the high overall precision achieved, current methods still suffer from a) imprecise object-level depth inference and b) uncertain scale factor. The former problem would cause texture copy or provide inaccurate object boundary, and the latter would require current methods to have an additional sensor like LiDAR to provide depth ground-truth or stereo camera as additional training inputs, which makes the…
Remarkable progress has been made in 3D human pose estimation from a monocular RGB camera. However, only a few studies explored 3D multi-person cases. In this paper, we attempt to address the lack of a global perspective of the top-down approaches by introducing a novel form of supervision - Hierarchical Multi-person Ordinal Relations (HMOR). The HMOR encodes interaction information as the ordinal relations of depths and angles hierarchically, which captures the body-part and joint level semantic and maintains global consistency at the same time. In our approach, an integrated top-down model i…
University of Freiburg（弗莱堡大学）；Università di Milano - Bicocca（米兰比可卡大学）
Localization is a critically essential and crucial enabler of autonomous robots. While deep learning has made significant strides in many computer vision tasks, it is still yet to make a sizeable impact on improving capabilities of metric visual localization. One of the major hindrances has been the inability of existing Convolutional Neural Network (CNN)-based pose regression methods to generalize to previously unseen places. Our recently introduced CMRNet effectively addresses this limitation by enabling map independent monocular localization in LiDAR-maps. In this paper, we now take it a st…
As an inherently ill-posed problem, depth estimation from single images is the most challenging part of monocular 3D object detection (M3OD). Many existing methods rely on preconceived assumptions to bridge the missing spatial information in monocular images, and predict a sole depth value for every object of interest. However, these assumptions do not always hold in practical applications. To tackle this problem, we propose a depth solving system that fully explores the visual clues from the subtasks in M3OD and generates multiple estimations for the depth of each target. Since the depth esti…
MonoCD: Monocular 3D Object Detection with Complementary Depths
Hubei Engineering Research Center of Machine Vision and Intelligent Systems（湖北省机器视觉与智能系统工程研究中心）；School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（华中科技大学人工智能与自动化学院）
GUPNet++: Geometry Uncertainty Propagation Network for Monocular 3D Object Detection
School of Electrical and Information Engineering, University of Sydney（悉尼大学电气与信息工程学院）；Shanghai AI Laboratory（上海人工智能实验室）；SenseTime Group Limited（商汤科技集团有限公司）；University of Science and Technology of China（中国科学技术大学）
PoseMoE: Mixture-of-Experts Network for Monocular 3D Human Pose Estimation
State Key Laboratory of General Artificial Intelligence, Peking University, Shenzhen Graduate School（国家通用人工智能重点实验室，北京大学深圳研究生院）；College of Computing and Data Science, Nanyang Technological University（计算与数据科学学院，南洋理工大学）；Department of Computer Science and Engineering, University at Buffalo, State University of New York（计算机科学与工程系，布法罗大学，纽约州立大学）
This paper presents an unsupervised deep learning framework called UnDEMoN for estimating dense depth map and 6-DoF camera pose information directly from monocular images. The proposed network is trained using unlabeled monocular stereo image pairs and is shown to provide superior performance in depth and ego-motion estimation compared to the existing state-of-the-art. These improvements are achieved by introducing a new objective function that aims to minimize spatial as well as temporal reconstruction losses simultaneously. These losses are defined using bi-linear sampling kernel and penaliz…
