Monocular depth estimation (MDE) has been widely adopted in the perception systems of autonomous vehicles and mobile robots. However, existing approaches often struggle to maintain temporal consistency in depth estimation across consecutive frames. This inconsistency not only causes jitter but can also lead to estimation failures when the depth range changes abruptly. To address these challenges, this paper proposes a consistency-aware monocular depth estimation framework that leverages wheel odometry from a mobile robot to achieve stable and coherent depth predictions over time. Specifically, we estimate camera pose and sparse depth from triangulation using optical flow between consecutive frames. The sparse depth estimates are used to update a recursive Bayesian estimate of the metric scale, which is then applied to rescale the relative depth predicted by a pre-trained depth estimation foundation model. The proposed method is evaluated on the KITTI, TartanAir, MS2, and our own dataset, demonstrating robust and accurate depth estimation performance.
In this paper, we strive for solving the ambiguities arisen by the astoundingly high density of raw PseudoLiDAR for monocular 3D object detection for autonomous driving. Without much computational overhead, we propose a supervised and an unsupervised sparsification scheme of PseudoLiDAR prior to 3D detection. Both the strategies assist the standard 3D detector gain better performance over the raw PseudoLiDAR baseline using only ~5% of its points on the KITTI object detection benchmark, thus making our monocular framework and LiDAR-based counterparts computationally equivalent (Figure 1). Moreo…
Monocular Depth Estimation (MDE) aims to predict pixel-wise depth given a single RGB image. For both, the convolutional as well as the recent attention-based models, encoder-decoder-based architectures have been found to be useful due to the simultaneous requirement of global context and pixel-level resolution. Typically, a skip connection module is used to fuse the encoder and decoder features, which comprises of feature map concatenation followed by a convolution operation. Inspired by the demonstrated benefits of attention in a multitude of computer vision problems, we propose an attention-…
Tianjin University（天津大学）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；The Chinese University of Hong Kong（香港中文大学）
In self-supervised monocular depth estimation, the depth discontinuity and motion objects' artifacts are still challenging problems. Existing self-supervised methods usually utilize a single view to train the depth estimation network. Compared with static views, abundant dynamic properties between video frames are beneficial to refined depth estimation, especially for dynamic objects. In this work, we propose a novel self-supervised joint learning framework for depth estimation using consecutive frames from monocular and stereo videos. The main idea is using an implicit depth cue extractor whi…
MonSter++: Unified Stereo Matching, Multi-view Stereo, and Real-time Stereo with Monodepth Priors
School of Electronic Information and Communications, Huazhong University of Science and Technology（电子信息与通讯学院，华中科技大学）；Meta；AI Chip Center, Hong Kong University of Science and Technology（香港科技大学人工智能芯片中心）；Nanjing Forestry University（南京林业大学）；Autel Robotics
Multi-frame depth estimation improves over single-frame approaches by also leveraging geometric relationships between images via feature matching, in addition to learning appearance-based features. In this paper we revisit feature matching for self-supervised monocular depth estimation, and propose a novel transformer architecture for cost volume generation. We use depth-discretized epipolar sampling to select matching candidates, and refine predictions through a series of self- and cross-attention layers. These layers sharpen the matching probability between pixel features, improving over sta…
Thermal Image Refinement with Depth Estimation using Recurrent Networks for Monocular ORB-SLAM3
Automatic Control Group (RAT), Paderborn University（自动控制组（RAT），波恩大学）；Department of Electrical and Computer Engineering, Aarhus University（电气与计算机工程系，阿arhus大学）；Upteko ApS（Upteko公司）
Due to the lack of depth information of images and poor detection accuracy in monocular 3D object detection, we proposed the instance depth for multi-scale monocular 3D object detection method. Firstly, to enhance the model's processing ability for different scale targets, a multi-scale perception module based on dilated convolution is designed, and the depth features containing multi-scale information are re-refined from both spatial and channel directions considering the inconsistency between feature maps of different scales. Firstly, we designed a multi-scale perception module based on dila…
Drone Stereo Vision for Radiata Pine Branch Detection and Distance Measurement: Utilizing Deep Learning and YOLO Integration
Centre for Data Science and Artificial Intelligence（数据科学与人工智能中心）；Victoria University of Wellington（惠灵顿维多利亚大学）；Department of Computer Science and Software Engineering（计算机科学与软件工程系）；Canterbury University（坎特伯雷大学）
University of Trento（特伦托大学）；Huawei Technologies Ireland（华为技术爱尔兰公司）；Fondazione Bruno Kessler（布鲁诺·凯斯勒基金会）
Nowadays, the majority of state of the art monocular depth estimation techniques are based on supervised deep learning models. However, collecting RGB images with associated depth maps is a very time consuming procedure. Therefore, recent works have proposed deep architectures for addressing the monocular depth prediction task as a reconstruction problem, thus avoiding the need of collecting ground-truth depth. Following these works, we propose a novel self-supervised deep model for estimating depth maps. Our framework exploits two main strategies: refinement via cycle-inconsistency and distil…
University of Washington（华盛顿大学）；Pacific States Marine Fisheries Commission（太平洋各州渔业委员会）；National Oceanic and Atmospheric Administration（美国国家海洋和大气管理局）
Monocular absolute 3D fish pose estimation allows for efficient fish length measurement in the longline fisheries, where fishes are under severe deformation during the catching process. This task is challenging since it requires locating absolute 3D fish keypoints based on a short monocular video clip. Unlike related works, which either require expensive 3D ground-truth data and/or multiple-view images to provide depth information, or are limited to rigid objects, we propose a novel frame-based method to estimate the absolute 3D fish pose and fish length from a single-view 2D segmentation mask…
Unifying Scale-Aware Depth Prediction and Perceptual Priors for Monocular Endoscope Pose Estimation and Tissue Reconstruction
The Robotics and Mechatronics Group, University of Twente（代尔夫嫩兰道5号，埃因霍温大学机器人与机电组）；The Department of Surgical Oncology, The Netherlands Cancer Institute-Antoni van Leeuwenhoek（荷兰癌症研究所-安托万·李文霍克医院外科肿瘤科）
RDG-GS: Relative Depth Guidance with Gaussian Splatting for Real-time Sparse-View 3D Rendering
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；College of Biomedical Engineering and Instrument Science, Zhejiang University（浙江大学生物医学工程与仪器科学学院）；ZJU-UIUC Institute, Zhejiang University（浙江大学-伊利诺伊大学厄巴纳-香槟分校联合学院）
This paper presents an online initialization method for bootstrapping the optimization-based monocular visual-inertial odometry (VIO). The method can online calibrate the relative transformation (spatial) and time offsets (temporal) among camera and IMU, as well as estimate the initial values of metric scale, velocity, gravity, gyroscope bias, and accelerometer bias during the initialization stage. To compensate for the impact of time offset, our method includes two short-term motion interpolation algorithms for the camera and IMU pose estimation. Besides, it includes a three-step process to i…
German Aerospace Center (DLR)（德国航空航天中心（DLR））；Technical University of Munich（慕尼黑工业大学）；ETH Zurich（苏黎世联邦理工学院）
In the future, extraterrestrial expeditions will not only be conducted by rovers but also by flying robots. The technical demonstration drone Ingenuity, that just landed on Mars, will mark the beginning of a new era of exploration unhindered by terrain traversability. Robust self-localization is crucial for that. Cameras that are lightweight, cheap and information-rich sensors are already used to estimate the ego-motion of vehicles. However, methods proven to work in man-made environments cannot simply be deployed on other planets. The highly repetitive textures present in the wastelands of Ma…
RaCalNet: Radar Calibration Network for Sparse-Supervised Metric Depth Estimation
School of Automation and Intelligent Sensing, Institute of Medical Robotics, Shanghai Jiao Tong University, Shanghai 200240, China（上海交通大学自动化与智能感知学院，医学机器人研究所）；School of Mathematical Sciences, Shanghai Jiao Tong University, Shanghai, China（上海交通大学数学科学学院）；South China University of Technology, Guangzhou, China（华南理工大学）；State Grid Intelligence Technology CO., LTD（国家电网智能技术有限公司）
PoseMoE: Mixture-of-Experts Network for Monocular 3D Human Pose Estimation
State Key Laboratory of General Artificial Intelligence, Peking University, Shenzhen Graduate School（国家通用人工智能重点实验室，北京大学深圳研究生院）；College of Computing and Data Science, Nanyang Technological University（计算与数据科学学院，南洋理工大学）；Department of Computer Science and Engineering, University at Buffalo, State University of New York（计算机科学与工程系，布法罗大学，纽约州立大学）
ChArUco boards are used for camera calibration, monocular pose estimation, and pose verification in both robotics and augmented reality. Such fiducials are detectable via traditional computer vision methods (as found in OpenCV) in well-lit environments, but classical methods fail when the lighting is poor or when the image undergoes extreme motion blur. We present Deep ChArUco, a real-time pose estimation system which combines two custom deep networks, ChArUcoNet and RefineNet, with the Perspective-n-Point (PnP) algorithm to estimate the marker's 6DoF pose. ChArUcoNet is a two-headed marker-sp…
Optical cameras are gaining popularity as the suitable sensor for relative navigation in space due to their attractive sizing, power and cost properties when compared to conventional flight hardware or costly laser-based systems. However, a camera cannot infer depth information on its own, which is often solved by introducing complementary sensors or a second camera. In this paper, an innovative model-based approach is instead demonstrated to estimate the six-dimensional pose of a target object relative to the chaser spacecraft using solely a monocular setup. The observed facet of the target i…
Accurate estimation of 3D human motion from monocular video requires modeling both kinematics (body motion without physical forces) and dynamics (motion with physical forces). To demonstrate this, we present SimPoE, a Simulation-based approach for 3D human Pose Estimation, which integrates image-based kinematic inference and physics-based dynamics modeling. SimPoE learns a policy that takes as input the current-frame pose estimate and the next image frame to control a physically-simulated character to output the next-frame pose estimate. The policy contains a learnable kinematic pose refinemen…
Visual Computing and Computational Media Section (VCCM)（视觉计算与计算媒体部门）；College of Performance Visualization and Fine Arts (PVFA)（表演可视化与美术学院）；Department of Electrical and Computer Engineering (ECEN)（电气与计算机工程系）；Department of Computer Science and Engineering (CSCE)（计算机科学与工程系）
