The problem of depth completion involves predicting a dense depth image from a single sparse depth map and an RGB image. Unsupervised depth completion methods have been proposed for various datasets where ground truth depth data is unavailable and supervised methods cannot be applied. However, these models require auxiliary data to estimate depth values, which is far from real scenarios. Monocular depth estimation (MDE) models can produce a plausible relative depth map from a single image, but there is no work to properly combine the sparse depth map with MDE for depth completion; a simple affine transformation to the depth map will yield a high error since MDE are inaccurate at estimating depth difference between objects. We introduce StarryGazer, a domain-agnostic framework that predicts dense depth images from a single sparse depth image and an RGB image without relying on ground-truth depth by leveraging the power of large MDE models. First, we employ a pre-trained MDE model to produce relative depth images. These images are segmented and randomly rescaled to form synthetic pairs for dense pseudo-ground truth and corresponding sparse depths. A refinement network is trained with the synthetic pairs, incorporating the relative depth maps and RGB images to improve the model's accuracy and robustness. StarryGazer shows superior results over existing unsupervised methods and transformed MDE results on various datasets, demonstrating that our framework exploits the power of MDE models while appropriately fixing errors using sparse depth information.
No Pose Estimation? No Problem: Pose-Agnostic and Instance-Aware Test-Time Adaptation for Monocular Depth Estimation
Department of Artificial Intelligence, Kyungpook National University（人工智能系，庆北国立大学）；Department of Electrical and Computer Engineering, Queen’s University（电气与计算机工程系，皇后大学）；Department of Information and Communications Engineering, Pukyong National University（信息与通信工程系，浦项国立大学）
Washington University in St. Louis（圣路易斯华盛顿大学）；The Boeing Company（波音公司）
Beyond depth estimation from a single image, the monocular cue is useful in a broader range of depth inference applications and settings---such as when one can leverage other available depth cues for improved accuracy. Currently, different applications, with different inference tasks and combinations of depth cues, are solved via different specialized networks---trained separately for each application. Instead, we propose a versatile task-agnostic monocular model that outputs a probability distribution over scene depth given an input color image, as a sample approximation of outputs from a pat…
University of Missouri（密苏里大学）；BOSCH Research China（博世中国研究中心）；BOSCH Research North America（博世北美研究中心）
Omnidirectional 3D information is essential for a wide range of applications such as Virtual Reality, Autonomous Driving, Robotics, etc. In this paper, we propose a novel, model-agnostic, two-stage pipeline for omnidirectional monocular depth estimation. Our proposed framework PanoDepth takes one 360 image as input, produces one or more synthesized views in the first stage, and feeds the original image and the synthesized images into the subsequent stereo matching stage. In the second stage, we propose a differentiable Spherical Warping Layer to handle omnidirectional stereo geometry efficient…
University of Oulu（奥卢大学）；Czech Technical University in Prague（布拉格捷克理工大学）；Tampere University（坦佩雷大学）
In this paper, we propose enhancing monocular depth estimation by adding 3D points as depth guidance. Unlike existing depth completion methods, our approach performs well on extremely sparse and unevenly distributed point clouds, which makes it agnostic to the source of the 3D points. We achieve this by introducing a novel multi-scale 3D point fusion network that is both lightweight and efficient. We demonstrate its versatility on two different depth estimation problems where the 3D points have been acquired with conventional structure-from-motion and LiDAR. In both cases, our network performs…
University of Southern California（南加州大学）；Meta Reality Labs（Meta现实实验室）
The majority of prior monocular depth estimation methods without groundtruth depth guidance focus on driving scenarios. We show that such methods generalize poorly to unseen complex indoor scenes, where objects are cluttered and arbitrarily arranged in the near field. To obtain more robustness, we propose a structure distillation approach to learn knacks from an off-the-shelf relative depth estimator that produces structured but metric-agnostic depth. By combining structure distillation with a branch that learns metrics from left-right consistency, we attain structured and metric depth for gen…
WildLIFT: Lifting monocular drone video to 3D for species-agnostic wildlife monitoring
Fondazione Bruno Kessler（布雷诺·科塞勒基金会）；University of Muenster（明斯特大学）；Max Planck Institute of Animal Behaviour（动物行为马克斯·普朗克研究所）；University of Konstanz（康斯坦茨大学）
Fluid-filled soft visuotactile sensors such as the Soft-bubbles alleviate key challenges for robust manipulation, as they enable reliable grasps along with the ability to obtain high-resolution sensory feedback on contact geometry and forces. Although they are simple in construction, their utility has been limited due to size constraints introduced by enclosed custom IR/depth imaging sensors to directly measure surface deformations. Towards mitigating this limitation, we investigate the application of state-of-the-art monocular depth estimation to infer dense internal (tactile) depth maps dire…
Stereo-Inertial Poser: Towards Metric-Accurate Shape-Aware Motion Capture Using Sparse IMUs and a Single Stereo Camera
School of Computer Science, Shanghai Jiao Tong University（上海交通大学计算机科学学院）；Meta Robotics Institute, SJTU（SJTU机器人研究院）；Shanghai Innovation Institute（上海创新研究院）
UNO: Unified Self-Supervised Monocular Odometry for Platform-Agnostic Deployment
Institute of Medical Robotics and Department of Automation, Shanghai Jiao Tong University（医学机器人研究所和自动化系，上海交通大学）；The Centre for Advanced Robotics Technology Innovation (CARTIN), School of Electrical and Electronic Engineering, Nanyang Technological University（先进机器人技术创新中心（CARTIN），电子与电气工程学院，南洋理工大学）；School of Electrical and Electronic Engineering, Nanyang Technological University（电子与电气工程学院，南洋理工大学）；Singapore State Grid Intelligence Technology CO., LTD.（新加坡国网智能技术有限公司）
DMS:Diffusion-Based Multi-Baseline Stereo Generation for Improving Self-Supervised Depth Estimation
Institute of Science Tokyo, Japan（东京科学研究所）；Sony Semiconductor Solutions Group, Japan（日本索尼半导体解决方案集团）；Nanyang Technological University, Singapore（南洋理工大学）
TopoCap: Learning Topology-Agnostic Motion Priors for Monocular Video-to-Animation
Zhili College, Tsinghua University（清华大学致理书院）；BNRist, Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系，北京国家信息科学与技术研究中心）；VAST
University of Freiburg（弗莱堡大学）；Università di Milano - Bicocca（米兰比可卡大学）
Localization is a critically essential and crucial enabler of autonomous robots. While deep learning has made significant strides in many computer vision tasks, it is still yet to make a sizeable impact on improving capabilities of metric visual localization. One of the major hindrances has been the inability of existing Convolutional Neural Network (CNN)-based pose regression methods to generalize to previously unseen places. Our recently introduced CMRNet effectively addresses this limitation by enabling map independent monocular localization in LiDAR-maps. In this paper, we now take it a st…
National University of Singapore（新加坡国立大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Saarland Informatics Campus（萨尔兰信息学园区）
We propose a novel approach to jointly perform 3D shape retrieval and pose estimation from monocular images.In order to make the method robust to real-world image variations, e.g. complex textures and backgrounds, we learn an embedding space from 3D data that only includes the relevant information, namely the shape and pose. Our approach explicitly disentangles a shape vector and a pose vector, which alleviates both pose bias for 3D shape retrieval and categorical bias for pose estimation. We then train a CNN to map the images to this embedding space, and then retrieve the closest 3D shape fro…
Tampere University（坦佩雷大学）；University of Oulu（奥卢大学）
This paper presents an efficient symmetry-agnostic and correspondence-free framework, referred to as SC6D, for 6D object pose estimation from a single monocular RGB image. SC6D requires neither the 3D CAD model of the object nor any prior knowledge of the symmetries. The pose estimation is decomposed into three sub-tasks: a) object 3D rotation representation learning and matching; b) estimation of the 2D location of the object center; and c) scale-invariant distance estimation (the translation along the z-axis) via classification. SC6D is evaluated on three benchmark datasets, T-LESS, YCB-V, a…
Valeo Egypt（法雷奥埃及）；University of Alberta（阿尔伯塔大学）；Valeo Ireland（法雷奥爱尔兰）；Zewail City of Science and Technology（扎威尔科技城）
Moving object segmentation is a crucial task for autonomous vehicles as it can be used to segment objects in a class agnostic manner based on their motion cues. It enables the detection of unseen objects during training (e.g., moose or a construction truck) based on their motion and independent of their appearance. Although pixel-wise motion segmentation has been studied in autonomous driving literature, it has been rarely addressed at the instance level, which would help separate connected segments of moving objects leading to better trajectory planning. As the main issue is the lack of large…
Deep neural networks have been widely studied in autonomous driving applications such as semantic segmentation or depth estimation. However, training a neural network in a supervised manner requires a large amount of annotated labels which are expensive and time-consuming to collect. Recent studies leverage synthetic data collected from a virtual environment which are much easier to acquire and more accurate compared to data from the real world, but they usually suffer from poor generalization due to the inherent domain shift problem. In this paper, we propose a Domain-Agnostic Contrastive Lea…
Detection of moving objects is a very important task in autonomous driving systems. After the perception phase, motion planning is typically performed in Bird's Eye View (BEV) space. This would require projection of objects detected on the image plane to top view BEV plane. Such a projection is prone to errors due to lack of depth information and noisy mapping in far away areas. CNNs can leverage the global context in the scene to project better. In this work, we explore end-to-end Moving Object Detection (MOD) on the BEV map directly using monocular images as input. To the best of our knowled…
We introduce the problem of perpetual view generation - long-range generation of novel views corresponding to an arbitrarily long camera trajectory given a single image. This is a challenging problem that goes far beyond the capabilities of current view synthesis methods, which quickly degenerate when presented with large camera motions. Methods for video generation also have limited ability to produce long sequences and are often agnostic to scene geometry. We take a hybrid approach that integrates both geometry and image synthesis in an iterative `\emph{render}, \emph{refine} and \emph{repea…
Any to Full: Prompting Depth Anything for Depth Completion in One Stage
Rutgers University（罗格斯大学）；Michigan State University（密歇根州立大学）；JD Logistics（京东物流）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
CurriFlow: Curriculum-Guided Depth Fusion with Optical Flow-Based Temporal Alignment for 3D Semantic Scene Completion
School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）；Ministry of Education（教育部）；Shandong Computer Science Center (National Supercomputer Center in Jinan)（山东省计算机科学中心（济南国家超算中心））；Qilu University of Technology (Shandong Academy of Sciences)（齐鲁工业大学（山东省科学院））；Shandong Provincial Key Laboratory of Computing Power Internet and Service Computing（山东省计算电力互联网与服务计算重点实验室）；Shandong Fundamental Research Center for Computer Science（山东省计算机科学基础研究中心）
