In recent times, monocular depth estimation (MDE) has experienced significant advancements in performance, largely attributed to the integration of innovative architectures, i.e., convolutional neural networks (CNNs) and Transformers. Nevertheless, the susceptibility of these models to adversarial attacks has emerged as a noteworthy concern, especially in domains where safety and security are paramount. This concern holds particular weight for MDE due to its critical role in applications like autonomous driving and robotic navigation, where accurate scene understanding is pivotal. To assess the vulnerability of CNN-based depth prediction methods, recent work tries to design adversarial patches against MDE. However, the existing approaches fall short of inducing a comprehensive and substantially disruptive impact on the vision system. Instead, their influence is partial and confined to specific local areas. These methods lead to erroneous depth predictions only within the overlapping region with the input image, without considering the characteristics of the target object, such as its size, shape, and position. In this paper, we introduce a novel adversarial patch named APARATE. This patch possesses the ability to selectively undermine MDE in two distinct ways: by distorting the estimated distances or by creating the illusion of an object disappearing from the perspective of the autonomous system. Notably, APARATE is designed to be sensitive to the shape and scale of the target object, and its influence extends beyond immediate proximity. APARATE, results in a mean depth estimation error surpassing $0.5$, significantly impacting as much as $99\%$ of the targeted region when applied to CNN-based MDE models. Furthermore, it yields a significant error of $0.34$ and exerts substantial influence over $94\%$ of the target region in the context of Transformer-based MDE.
Graduate School of Engineering, Nagoya University（名古屋大学大学院工学研究科）；Tokyo Institute of Technology（东京工业大学）；Aalborg University（奥尔堡大学）
Thanks to the excellent learning capability of deep convolutional neural networks (CNN), monocular depth estimation using CNNs has achieved great success in recent years. However, depth estimation from a monocular image alone is essentially an ill-posed problem, and thus, it seems that this approach would have inherent vulnerabilities. To reveal this limitation, we propose a method of adversarial patch attack on monocular depth estimation. More specifically, we generate artificial patterns (adversarial patches) that can fool the target methods into estimating an incorrect depth for the regions…
Purdue University（普渡大学）；Rochester Institute of Technology（罗切斯特理工学院）
Deep learning has substantially boosted the performance of Monocular Depth Estimation (MDE), a critical component in fully vision-based autonomous driving (AD) systems (e.g., Tesla and Toyota). In this work, we develop an attack against learning-based MDE. In particular, we use an optimization-based method to systematically generate stealthy physical-object-oriented adversarial patches to attack depth estimation. We balance the stealth and effectiveness of our attack with object-oriented adversarial design, sensitive region localization, and natural style camouflage. Using real-world driving s…
Bosch Research North America（博世北美研究院）；University of Missouri（密苏里大学）
A well-known challenge in applying deep-learning methods to omnidirectional images is spherical distortion. In dense regression tasks such as depth estimation, where structural details are required, using a vanilla CNN layer on the distorted 360 image results in undesired information loss. In this paper, we propose a 360 monocular depth estimation pipeline, OmniFusion, to tackle the spherical distortion issue. Our pipeline transforms a 360 image into less-distorted perspective patches (i.e. tangent images) to obtain patch-wise predictions via CNN, and then merge the patch-wise results for fina…
Bournemouth University（伯恩茅斯大学）；University of Chester（切斯特大学）
Convolutional Neural Networks (CNNs) need large amounts of data with ground truth annotation, which is a challenging problem that has limited the development and fast deployment of CNNs for many computer vision tasks. We propose a novel framework for depth estimation from monocular images with corresponding confidence in a self-supervised manner. A fully differential patch-based cost function is proposed by using the Zero-Mean Normalized Cross Correlation (ZNCC) that takes multi-scale patches as a matching strategy. This approach greatly increases the accuracy and robustness of the depth learn…
Monocular depth estimation is an especially important task in robotics and autonomous driving, where 3D structural information is essential. However, extreme lighting conditions and complex surface objects make it difficult to predict depth in a single image. Therefore, to generate accurate depth maps, it is important for the model to learn structural information about the scene. We propose a novel Patch-Wise EdgeConv Module (PEM) and EdgeConv Attention Module (EAM) to solve the difficulty of monocular depth estimation. The proposed modules extract structural information by learning the relati…
Mila - Quebec AI Institute（米拉-魁北克人工智能研究所）；Université de Montréal（蒙特利尔大学）
In this work, we consider the problem of learning a perception model for monocular robot navigation using few annotated images. Using a Vision Transformer (ViT) pretrained with a label-free self-supervised method, we successfully train a coarse image segmentation model for the Duckietown environment using 70 training images. Our model performs coarse image segmentation at the 8x8 patch level, and the inference resolution can be adjusted to balance prediction granularity and real-time perception constraints. We study how best to adapt a ViT to our task and environment, and find that some lightw…
Washington University in St. Louis（圣路易斯华盛顿大学）；The Boeing Company（波音公司）
Beyond depth estimation from a single image, the monocular cue is useful in a broader range of depth inference applications and settings---such as when one can leverage other available depth cues for improved accuracy. Currently, different applications, with different inference tasks and combinations of depth cues, are solved via different specialized networks---trained separately for each application. Instead, we propose a versatile task-agnostic monocular model that outputs a probability distribution over scene depth given an input color image, as a sample approximation of outputs from a pat…
We present PHORHUM, a novel, end-to-end trainable, deep neural network methodology for photorealistic 3D human reconstruction given just a monocular RGB image. Our pixel-aligned method estimates detailed 3D geometry and, for the first time, the unshaded surface color together with the scene illumination. Observing that 3D supervision alone is not sufficient for high fidelity color reconstruction, we introduce patch-based rendering losses that enable reliable color reconstruction on visible parts of the human, and detailed and plausible color estimation for the non-visible parts. Moreover, our …
$\mathrm{F^2Depth}$: Self-supervised Indoor Monocular Depth Estimation via Optical Flow Consistency and Feature Map Synthesis
Institute of Artificial Intelligence, Beihang University（北京航空航天大学人工智能研究院）；Qingdao Research Institute of Beihang University（北京航空航天大学青岛研究院）；School of Automation Science and Electrical Engineering, Beihang University（北京航空航天大学自动化科学与电气工程学院）
Recent advancements of neural networks lead to reliable monocular depth estimation. Monocular depth estimated techniques have the upper hand over traditional depth estimation techniques as it only needs one image during inference. Depth estimation is one of the essential tasks in robotics, and monocular depth estimation has a wide variety of safety-critical applications like in self-driving cars and surgical devices. Thus, the robustness of such techniques is very crucial. It has been shown in recent works that these deep neural networks are highly vulnerable to adversarial samples for tasks l…
University of Parma（帕尔马大学）；VisLab Srl（VisLab有限责任公司）；Ambarella Inc（安霸公司）
In this paper, a complete pipeline for image-based 3D reconstruction of urban scenarios is proposed, based on PatchMatch Multi-View Stereo (MVS). Input images are firstly fed into an off-the-shelf visual SLAM system to extract camera poses and sparse keypoints, which are used to initialize PatchMatch optimization. Then, pixelwise depths and normals are iteratively computed in a multi-scale framework with a novel depth-normal consistency loss term and a global refinement algorithm to balance the inherently local nature of PatchMatch. Finally, a large-scale point cloud is generated by back-proje…
Monocular depth estimation is an extensively studied computer vision problem with a vast variety of applications. Deep learning-based methods have demonstrated promise for both supervised and unsupervised depth estimation from monocular images. Most existing approaches treat depth estimation as a regression problem with a local pixel-wise loss function. In this work, we innovate beyond existing approaches by using adversarial training to learn a context-aware, non-local loss function. Such an approach penalizes the joint configuration of predicted depth values at the patch-level instead of the…
University of Surrey（萨里大学）；Aston University（阿斯顿大学）；Amazon（亚马逊公司）；University of Southampton（南安普顿大学）；York University（约克大学）；University of Bologna（博洛尼亚大学）；DENSO ADAS EngineeringServices GmbH（电装高级驾驶辅助系统工程服务有限公司）；East China University of Science and Technology（华东理工大学）
This paper summarizes the results of the first Monocular Depth Estimation Challenge (MDEC) organized at WACV2023. This challenge evaluated the progress of self-supervised monocular depth estimation on the challenging SYNS-Patches dataset. The challenge was organized on CodaLab and received submissions from 4 valid teams. Participants were provided a devkit containing updated reference implementations for 16 State-of-the-Art algorithms and 4 novel techniques. The threshold for acceptance for novel techniques was to outperform every one of the 16 SotA baselines. All participants outperformed the…
In this work, we propose a monocular visual odometry framework, which allows exploiting the best attributes of edge feature for illumination-robust camera tracking, while at the same time ameliorating the performance degradation of edge mapping. In the front-end, an ICP-based edge registration can provide robust motion estimation and coarse data association under lighting changes. In the back-end, a novel edge-guided data association pipeline searches for the best photometrically matched points along geometrically possible edges through template matching, so that the matches can be further ref…
Bahcesehir University（巴赫切谢希尔大学）；American University of Malta（马耳他美国大学）；Multimedia University（多媒体大学）
Monocular omnidirectional depth estimation is receiving considerable research attention due to its broad applications for sensing 360° surroundings. Existing approaches in this field suffer from limitations in recovering small object details and data lost during the ground-truth depth map acquisition. In this paper, a novel monocular omnidirectional depth estimation model, namely HiMODE is proposed based on a hybrid CNN+Transformer (encoder-decoder) architecture whose modules are efficiently designed to mitigate distortion and computational cost, without performance degradation. Firstly, we de…
Simon Fraser University（西蒙菲莎大学）；Adobe Research（Adobe研究院）
Neural networks have shown great abilities in estimating depth from a single image. However, the inferred depth maps are well below one-megapixel resolution and often lack fine-grained details, which limits their practicality. Our method builds on our analysis on how the input resolution and the scene structure affects depth estimation performance. We demonstrate that there is a trade-off between a consistent scene structure and the high-frequency details, and merge low- and high-resolution estimations to take advantage of this duality using a simple depth merging network. We present a double …
In this article, we tackle the problem of depth estimation from single monocular images. Compared with depth estimation using multiple images such as stereo depth perception, depth from monocular images is much more challenging. Prior work typically focuses on exploiting geometric priors or additional sources of information, most using hand-crafted features. Recently, there is mounting evidence that features from deep convolutional neural networks (CNN) set new records for various vision applications. On the other hand, considering the continuous characteristic of the depth values, depth estim…
Center for Robotics, University of Bonn（博恩大学机器人中心）；MAVLab, TU Delft（代尔夫特理工大学MAVLab）；Lamarr Institute for Machine Learning and Artificial Intelligence（兰玛尔人工智能与机器学习研究所）
TATA Consultancy Services（塔塔咨询服务公司）；Queensland University of Technology（昆士兰科技大学）；Edge Hill University（边山大学）
In this paper, we look into the problem of estimating per-pixel depth maps from unconstrained RGB monocular night-time images which is a difficult task that has not been addressed adequately in the literature. The state-of-the-art day-time depth estimation methods fail miserably when tested with night-time images due to a large domain shift between them. The usual photo metric losses used for training these networks may not work for night-time images due to the absence of uniform lighting which is commonly present in day-time images, making it a difficult problem to solve. We propose to solve …
Peking University（北京大学）；Advanced Institute of Information Technology, Peking University（北京大学先进信息技术研究院）；Information Technology R&D Innovation Center of Peking University（北京大学信息技术研发创新中心）
Self-supervised monocular depth estimation (MDE) models universally suffer from the notorious edge-fattening issue. Triplet loss, as a widespread metric learning strategy, has largely succeeded in many computer vision applications. In this paper, we redesign the patch-based triplet loss in MDE to alleviate the ubiquitous edge-fattening issue. We show two drawbacks of the raw triplet loss in MDE and demonstrate our problem-driven redesigns. First, we present a min. operator based strategy applied to all negative samples, to prevent well-performing negatives sheltering the error of edge-fattenin…
SCE-SLAM: Scale-Consistent Monocular SLAM via Scene Coordinate Embeddings
School of Computer Science and Engineering, State Key Laboratory of Complex & Critical Software Environment, Jiangxi Research Institute, Beihang University（计算机科学与工程学院、复杂与关键软件环境国家重点实验室、江西研究院、北航）；Macquarie University（麦考瑞大学）；Beijing Key Laboratory of Semiconductor Neural Network Intelligent Sensing and Computing Technology（北京半导体神经网络智能感知与计算技术重点实验室）
University of Bologna（博洛尼亚大学）；Blue River Technology（蓝河科技）；Oxford Internet Institute（牛津互联网研究院）；University of Surrey（萨里大学）；ByteDance（字节跳动）；University of Chinese Academy of Science（中国科学院大学）；RGA Inc.（RGA公司）；Space Research Institute NASU-SSAU（乌克兰国家科学院-空间研究协会空间研究所）；Northwestern Polytechnical University（西北工业大学）；Indian Institute of Technology, Delhi（印度理工学院德里分校）；Harbin Institute of Technology（哈尔滨工业大学）；Fujitsu（富士通）；GuangXi University（广西大学）；University of Reading（雷丁大学）；York University（约克大学）
University of Bologna（博洛尼亚大学）；Oxford Internet Institute（牛津互联网研究院）；University of Surrey（萨里大学）；Hikvision Research Institute（海康威视研究院）；Megvii（旷视科技）；Insta360（影石创新）；Harbin Institute of Technology（哈尔滨工业大学）；University of Trento（特伦托大学）；University of Science, VNU-HCM（胡志明市越南科学大学）；University of Reading（雷丁大学）
Hybrid-grained Feature Aggregation with Coarse-to-fine Language Guidance for Self-supervised Monocular Depth Estimation
MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University（人工智能 MOE 实验室，上海交通大学人工智能学院）；Ningbo Institute of Digital Twin, Eastern Institute of Technology, Ningbo, China（宁波数字孪生研究所，东技术研究所，宁波，中国）；Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative, Ningbo, China（宁波空间智能与数字衍生关键实验室，宁波，中国）；University of Science and Technology of China（中国科学技术大学）；CASIA；Tsinghua University（清华大学）
Peking University（北京大学）；ShanghaiTech University（上海科技大学）；Baoji University of Arts and Science（宝鸡文理学院）
Besides image classification, Contrastive Language-Image Pre-training (CLIP) has accomplished extraordinary success for a wide range of vision tasks, including object-level and 3D space understanding. However, it's still challenging to transfer semantic knowledge learned from CLIP into more intricate tasks of quantified targets, such as depth estimation with geometric information. In this paper, we propose to apply CLIP for zero-shot monocular depth estimation, named DepthCLIP. We found that the patches of the input image could respond to a certain semantic distance token and then be projected…
University of Surrey（萨里大学）；Aston University（阿斯顿大学）；University of Southampton（南安普顿大学）；Amazon（亚马逊）；York University（约克大学）；imec-University of Antwerp（比利时微电子研究中心-安特卫普大学）；Zhejiang University（浙江大学）；DJI Technology（大疆创新科技）；University of Science and Technology of China（中国科学技术大学）；Northwestern Polytechnical University（西北工业大学）；DeltaX（DeltaX公司）；VIVO（维沃）；University of Bologna（博洛尼亚大学）；East China University of Science and Technology（华东理工大学）
