BoRe-Depth: Self-supervised Monocular Depth Estimation with Boundary Refinement for Embedded Systems
School of Mechatronical Engineering, Beijing Institute of Technology（机械电子工程学院，北京理工大学）；Yangtze Delta Region Academy of Beijing Institute of Technology Jiaxing, China（北京理工大学嘉兴 Yangtze Delta Region 学院）
Chang Liu, Juan Li, Sheng Zhang, Chang Liu, Jie Li, Xu Zhang
Depth estimation is one of the key technologies for realizing 3D perception in unmanned systems. Monocular depth estimation has been widely researched because of its low-cost advantage, but the existing methods face the challenges of poor depth estimation performance and blurred object boundaries on embedded systems. In this paper, we propose a novel monocular depth estimation model, BoRe-Depth, which contains only 8.7M parameters. It can accurately estimate depth maps on embedded systems and significantly improves boundary quality. Firstly, we design an Enhanced Feature Adaptive Fusion Module (EFAF) which adaptively fuses depth features to enhance boundary detail representation. Secondly, we integrate semantic knowledge into the encoder to improve the object recognition and boundary perception capabilities. Finally, BoRe-Depth is deployed on NVIDIA Jetson Orin, and runs efficiently at 50.7 FPS. We demonstrate that the proposed model significantly outperforms previous lightweight models on multiple challenging datasets, and we provide detailed ablation studies for the proposed methods. The code is available at https://github.com/liangxiansheng093/BoRe-Depth.
Monocular depth estimation is the task of obtaining a measure of distance for each pixel using a single image. It is an important problem in computer vision and is usually solved using neural networks. Though recent works in this area have shown significant improvement in accuracy, the state-of-the-art methods tend to require massive amounts of memory and time to process an image. The main purpose of this work is to improve the performance of the latest solutions with no decrease in accuracy. To this end, we introduce the Double Refinement Network architecture. The proposed method achieves sta…
The Chinese University of Hong Kong（香港中文大学）；Nanyang Technological University（南洋理工大学）
3D interacting hand reconstruction is essential to facilitate human-machine interaction and human behaviors understanding. Previous works in this field either rely on auxiliary inputs such as depth images or they can only handle a single hand if monocular single RGB images are used. Single-hand methods tend to generate collided hand meshes, when applied to closely interacting hands, since they cannot model the interactions between two hands explicitly. In this paper, we make the first attempt to reconstruct 3D interacting hands from monocular single RGB images. Our method can generate 3D hand …
Thermal Image Refinement with Depth Estimation using Recurrent Networks for Monocular ORB-SLAM3
Automatic Control Group (RAT), Paderborn University（自动控制组（RAT），波恩大学）；Department of Electrical and Computer Engineering, Aarhus University（电气与计算机工程系，阿arhus大学）；Upteko ApS（Upteko公司）
Deep-learning-based approaches to depth estimation are rapidly advancing, offering superior performance over existing methods. To estimate the depth in real-world scenarios, depth estimation models require the robustness of various noise environments. In this work, a Pyramid Frequency Network(PFN) with Spatial Attention Residual Refinement Module(SARRM) is proposed to deal with the weak robustness of existing deep-learning methods. To reconstruct depth maps with accurate details, the SARRM constructs a residual fusion method with an attention mechanism to refine the blur depth. The frequency d…
Monocular Depth Guided Occlusion-Aware Disparity Refinement via Semi-supervised Learning in Laparoscopic Images
School of Life Science and Technology, Harbin Institute of Technology（生命科学与技术学院，哈尔滨工业大学）；State Key Laboratory of Robotics and System, Harbin Institute of Technology（机器人与系统国家重点实验室，哈尔滨工业大学）
PhysDepth: Plug-and-Play Physical Refinement for Monocular Depth Estimation in Challenging Environments
Department of Computer Science, East Carolina University（东卡罗来纳大学计算机科学系）；Department of Electrical and Computer Engineering, The University of Arizona（亚利桑那大学电气与计算机工程系）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络安全学院）；School of Computer and Cyber Sciences, Augusta University（奥古斯塔大学计算机与网络安全学院）；VCIP, CS, Nankai University（南开大学）
School of Mechanical and Robotics Engineering, Gwangju Institute of Science and Technology (GIST)（光州科学技术学院机械与机器人工程学院）；Department of Smart ICT Convergence Engineering, Seoul National University of Science and Technology（首尔科学技术大学智能ICT融合工程系）
SA4Depth: Consistent Pose-Depth Scale Alignment for Self-Supervised Monocular Depth Estimation
Technical University of Munich（慕尼黑技术大学）；BMW Group（宝马集团）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心（MCML））；Google（谷歌）；VisualAIs Labs GmbH（VisualAIs实验室 GmbH）
We present animatable neural radiance fields (animatable NeRF) for detailed human avatar creation from monocular videos. Our approach extends neural radiance fields (NeRF) to the dynamic scenes with human movements via introducing explicit pose-guided deformation while learning the scene representation network. In particular, we estimate the human pose for each frame and learn a constant canonical space for the detailed human template, which enables natural shape deformation from the observation space to the canonical space under the explicit control of the pose parameters. To compensate for i…
We present a robotic grasping system that uses a single external monocular RGB camera as input. The object-to-robot pose is computed indirectly by combining the output of two neural networks: one that estimates the object-to-camera pose, and another that estimates the robot-to-camera pose. Both networks are trained entirely on synthetic data, relying on domain randomization to bridge the sim-to-real gap. Because the latter network performs online camera calibration, the camera can be moved freely during execution without affecting the quality of the grasp. Experimental results analyze the effe…
Acquiring accurate three-dimensional depth information conventionally requires expensive multibeam LiDAR devices. Recently, researchers have developed a less expensive option by predicting depth information from two-dimensional color imagery. However, there still exists a substantial gap in accuracy between depth information estimated from two-dimensional images and real LiDAR point-cloud. In this paper, we introduce a fusion-based depth prediction method, called FusionMapping. This is the first method that fuses colored imagery and two-dimensional laser scan to estimate depth in-formation. Mo…
Self-supervised monocular depth estimation presents a powerful method to obtain 3D scene information from single camera images, which is trainable on arbitrary image sequences without requiring depth labels, e.g., from a LiDAR sensor. In this work we present a new self-supervised semantically-guided depth estimation (SGDepth) method to deal with moving dynamic-class (DC) objects, such as moving cars and pedestrians, which violate the static-world assumptions typically made during training of such models. Specifically, we propose (i) mutually beneficial cross-domain training of (supervised) sem…
Recent advances in end-to-end unsupervised learning has significantly improved the performance of monocular depth prediction and alleviated the requirement of ground truth depth. Although a plethora of work has been done in enforcing various structural constraints by incorporating multiple losses utilising smoothness, left-right consistency, regularisation and matching surface normals, a few of them take into consideration multi-scale structures present in real world images. Most works utilise a VGG16 or ResNet50 model pre-trained on ImageNet weights for predicting depth. We propose a deep fea…
Beihang University（北京航空航天大学）；Hangzhou Innovation Institute, Beihang University（北京航空航天大学杭州创新研究院）；Institute for Infocomm Research, A*STAR（新加坡科技研究局资讯通信研究院）
Niantic（Niantic公司）；University of Edinburgh（爱丁堡大学）；University of Oxford（牛津大学）；UCL（伦敦大学学院）
Self-supervised monocular depth estimation networks are trained to predict scene depth using nearby frames as a supervision signal during training. However, for many applications, sequence information in the form of video frames is also available at test time. The vast majority of monocular networks do not make use of this extra signal, thus ignoring valuable information that could be used to improve the predicted depth. Those that do, either use computationally expensive test-time refinement techniques or off-the-shelf recurrent networks, which only indirectly make use of the geometric inform…
Bosch Research North America（博世北美研究院）；University of Missouri（密苏里大学）
A well-known challenge in applying deep-learning methods to omnidirectional images is spherical distortion. In dense regression tasks such as depth estimation, where structural details are required, using a vanilla CNN layer on the distorted 360 image results in undesired information loss. In this paper, we propose a 360 monocular depth estimation pipeline, OmniFusion, to tackle the spherical distortion issue. Our pipeline transforms a 360 image into less-distorted perspective patches (i.e. tangent images) to obtain patch-wise predictions via CNN, and then merge the patch-wise results for fina…
Ground4D: Consistency-Aware 4D Reconstruction from Monocular Video
School of Computer Science and Engineering, Sun Yat-sen University, China（中山大学计算机科学与工程学院）；Tsinghua Shenzhen International Graduate School, Tsinghua University, China（清华大学深圳国际研究生院）
Single image depth estimation is a challenging problem. The current state-of-the-art method formulates the problem as that of ordinal regression. However, the formulation is not fully differentiable and depth maps are not generated in an end-to-end fashion. The method uses a naïve threshold strategy to determine per-pixel depth labels, which results in significant discretization errors. For the first time, we formulate a fully differentiable ordinal regression and train the network in end-to-end fashion. This enables us to include boundary and smoothness constraints in the optimization functio…
M^3: Dense Matching Meets Multi-View Foundation Models for Monocular Gaussian Splatting SLAM
Shanghai Jiao Tong University（上海交通大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Fudan University（复旦大学）；Shanghai Innovation Institute（上海创新研究院）；Zhejiang University（浙江大学）；Beijing Institute of Technology（北京理工大学）；The Chinese University of Hong Kong（香港中文大学）；The University of Hong Kong（香港大学）
National Engineering Laboratory for Brain-inspired Intelligence Technology and Application（类脑智能技术与应用国家工程实验室）；University of Science and Technology of China（中国科学技术大学）
Monocular depth estimation is an essential task for scene understanding. The underlying structure of objects and stuff in a complex scene is critical to recovering accurate and visually-pleasing depth maps. Global structure conveys scene layouts, while local structure reflects shape details. Recently developed approaches based on convolutional neural networks (CNNs) significantly improve the performance of depth estimation. However, few of them take into account multi-scale structures in complex scenes. In this paper, we propose a Structure-Aware Residual Pyramid Network (SARPN) to exploit mul…
Monocular Depth Estimation and Segmentation for Transparent Object with Iterative Semantic and Geometric Fusion
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Institute of Automation of Chinese Academy of Sciences（中国科学院自动化研究所）；School of Information Science and Technology, Shijiazhuang Tiedao University（石家庄铁道大学信息科学与技术学院）；D-Robotics（杭州宇树科技有限公司）
Monocular Depth Estimation (MDE) aims to predict pixel-wise depth given a single RGB image. For both, the convolutional as well as the recent attention-based models, encoder-decoder-based architectures have been found to be useful due to the simultaneous requirement of global context and pixel-level resolution. Typically, a skip connection module is used to fuse the encoder and decoder features, which comprises of feature map concatenation followed by a convolution operation. Inspired by the demonstrated benefits of attention in a multitude of computer vision problems, we propose an attention-…
Robust 3D Alignment of Generative Reconstructions via Partial Monocular Observations
School of Computer Science, Shanghai Jiao Tong University（上海交通大学计算机科学与技术学院）；School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；School of Engineering and Applied Science, University of Pennsylvania（宾夕法尼亚大学工程与应用科学学院）；School of Tech Electrical and Information Engineering, Tianjin University（天津大学电气与信息工程学院）；Independent Researcher（独立研究员）；School of Electronic Engineering and Information Science, University of Science and Technology of China（中国科学技术大学电子工程与信息科学学院）；Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；School of Robotics, Wuhan University（武汉大学机器人学院）；Xiaomi EV（小米汽车）
Bidirectional Regression for Monocular 6DoF Head Pose Estimation and Reference System Alignment
Department of Electronics and Communications Engineering, Kwangwoon University（韩国 Kwangwoon 大学电子与通信工程系）；Department of Software Engineering, Dankook University（韩国 Dankook 大学软件工程系）；School of Computer Science, University of Birmingham（英国 Birmingham 大学计算机科学学院）
Beijing University of Posts and Telecommunications（北京邮电大学）；Huazhong University of Science and Technology（华中科技大学）
Monocular depth prediction is an important task in scene understanding. It aims to predict the dense depth of a single RGB image. With the development of deep learning, the performance of this task has made great improvements. However, two issues remain unresolved: (1) The deep feature encodes the wrong farthest region in a scene, which leads to a distorted 3D structure of the predicted depth; (2) The low-level features are insufficient utilized, which makes it even harder to estimate the depth near the edge with sudden depth change. To tackle these two issues, we propose the Boundary-induced …
ARDepth: Auto-regressive Monocular Depth Estimation with Progressive Visual Conditioning
School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；Shenzhen Loop Area Institute（深圳河套学院）；Guangdong Key Laboratory of Big Data Analysis and Processing（广东省大数据分析与处理重点实验室）；Baidu Inc.（百度公司）；LightSpeed Studios, Tencent America（美国光速工作室，腾讯）；School of Computer Science and Artificial Intelligence, Lanzhou University of Technology（兰州理工大学计算机科学与人工智能学院）
