Autonomous cars need continuously updated depth information. Thus far, depth is mostly estimated independently for a single frame at a time, even if the method starts from video input. Our method produces a time series of depth maps, which makes it an ideal candidate for online learning approaches. In particular, we put three different types of depth estimation (supervised depth prediction, self-supervised depth prediction, and self-supervised depth completion) into a common framework. We integrate the corresponding networks with a ConvLSTM such that the spatiotemporal structures of depth across frames can be exploited to yield a more accurate depth estimation. Our method is flexible. It can be applied to monocular videos only or be combined with different types of sparse depth patterns. We carefully study the architecture of the recurrent network and its training strategy. We are first to successfully exploit recurrent networks for real-time self-supervised monocular depth estimation and completion. Extensive experiments show that our recurrent method outperforms its image-based counterpart consistently and significantly in both self-supervised scenarios. It also outperforms previous depth estimation methods of the three popular groups. Please refer to https://www.trace.ethz.ch/publications/2020/rec_depth_estimation/ for details.
This paper aims at understanding the role of multi-scale information in the estimation of depth from monocular images. More precisely, the paper investigates four different deep CNN architectures, designed to explicitly make use of multi-scale features along the network, and compare them to a state-of-the-art single-scale approach. The paper also shows that involving multi-scale features in depth estimation not only improves the performance in terms of accuracy, but also gives qualitatively better depth maps. Experiments are done on the widely used NYU Depth dataset, on which the proposed meth…
Dense depth estimation from a single image is a key problem in computer vision, with exciting applications in a multitude of robotic tasks. Initially viewed as a direct regression problem, requiring annotated labels as supervision at training time, in the past few years a substantial amount of work has been done in self-supervised depth training based on strong geometric cues, both from stereo cameras and more recently from monocular video sequences. In this paper we investigate how these two approaches (supervised & self-supervised) can be effectively combined, so that a depth model can learn…
Recent advances in end-to-end unsupervised learning has significantly improved the performance of monocular depth prediction and alleviated the requirement of ground truth depth. Although a plethora of work has been done in enforcing various structural constraints by incorporating multiple losses utilising smoothness, left-right consistency, regularisation and matching surface normals, a few of them take into consideration multi-scale structures present in real world images. Most works utilise a VGG16 or ResNet50 model pre-trained on ImageNet weights for predicting depth. We propose a deep fea…
An active-learning framework for real-time depth perception from monocular vision streams
School of Mechanical and Automotive Engineering, Xiamen University of Technology（厦门理工学院机械与汽车工程学院）；Xiamen Innovative Centre for Automotive Electric Driving（厦门新能源汽车驱动创新中心）；King Long United Automotive Industry Co. Ltd.（金龙联合汽车工业有限公司）；School of Electrical and Electronic Engineering, The University of Adelaide（阿德莱德大学电气与电子工程学院）；School of Vehicle and Mobility, Tsinghua University（清华大学车辆与运载学院）
SA4Depth: Consistent Pose-Depth Scale Alignment for Self-Supervised Monocular Depth Estimation
Technical University of Munich（慕尼黑技术大学）；BMW Group（宝马集团）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心（MCML））；Google（谷歌）；VisualAIs Labs GmbH（VisualAIs实验室 GmbH）
Centre for Research and Technology Hellas（希腊研究与技术中心）；Universidad Politécnica de Madrid（马德里理工大学）
Spherical cameras capture scenes in a holistic manner and have been used for room layout estimation. Recently, with the availability of appropriate datasets, there has also been progress in depth estimation from a single omnidirectional image. While these two tasks are complementary, few works have been able to explore them in parallel to advance indoor geometric perception, and those that have done so either relied on synthetic data, or used small scale datasets, as few options are available that include both layout annotations and dense depth maps in real scenes. This is partly due to the ne…
ENSTA ParisTech（巴黎高等先进技术学院）；INRIA Flowers（法国国家信息与自动化研究所Flowers实验室）；University of Bologna（博洛尼亚大学）
Continual learning consists of algorithms that learn from a stream of data/tasks continuously and adaptively thought time, enabling the incremental development of ever more complex knowledge and skills. The lack of consensus in evaluating continual learning algorithms and the almost exclusive focus on forgetting motivate us to propose a more comprehensive set of implementation independent metrics accounting for several factors we believe have practical implications worth considering in the deployment of real AI systems that learn continually: accuracy or performance over time, backward and for…
DARES: Depth Anything in Robotic Endoscopic Surgery with Self-supervised Vector-LoRA of the Foundation Model
Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS)（惠康/英国工程与物理科学研究委员会介入与外科科学中心（WEISS））；University College London（伦敦大学学院）；Politecnico di Milano（米兰理工大学）；Centre for Medical Image Computing（医学影像计算中心）
South China University of Technology（华南理工大学）；Pazhou Laboratory (Huangpu)（琶洲实验室（黄埔））；Peng Cheng Laboratory（鹏城实验室）；Huazhong University of Science and Technology（华中科技大学）；IntSig Information Co. Ltd（合合信息科技股份有限公司）
Text removal has attracted increasingly attention due to its various applications on privacy protection, document restoration, and text editing. It has shown significant progress with deep neural network. However, most of the existing methods often generate inconsistent results for complex background. To address this issue, we propose a Contextual-guided Text Removal Network, termed as CTRNet. CTRNet explores both low-level structure and high-level discriminative context feature as prior knowledge to guide the process of background restoration. We further propose a Local-global Content Modelin…
The University of Adelaide（阿德莱德大学）；Monash University（莫纳什大学）
Monocular depth prediction plays a crucial role in understanding 3D scene geometry. Although recent methods have achieved impressive progress in terms of evaluation metrics such as the pixel-wise relative error, most methods neglect the geometric constraints in the 3D space. In this work, we show the importance of the high-order 3D geometric constraints for depth prediction. By designing a loss term that enforces a simple geometric constraint, namely, virtual normal directions determined by randomly sampled three points in the reconstructed 3D space, we significantly improve the accuracy and r…
HDR-NSFF: High Dynamic Range Neural Scene Flow Fields
School of Electrical Engineering, KAIST（韩国科学技术院电子工程学院）；Dept. of Electrical Engineering, POSTECH（POSTECH电子工程系）；Grad. School of Artificial Intelligence, POSTECH（POSTECH人工智能研究生院）；School of Computing, KAIST（韩国科学技术院计算机科学学院）
University of Granada and DaSCI Research Institute（格拉纳达大学和DaSCI研究机构）；Panacea Cooperative Research（Panacea合作研究）；Inria at University Grenoble Alpes（Grenoble Alpes大学的Inria）
本文提出 SAGE 技术，利用预训练扩散模型的自注意力层和逆 DDIM 过程实现高效的真实图像编辑，在定量分析和用户研究中表现优于其他方法。
The University of Adelaide（阿德莱德大学）；Noah’s Ark Lab, Huawei Technologies（华为技术诺亚方舟实验室）
Monocular depth prediction plays a crucial role in understanding 3D scene geometry. Although recent methods have achieved impressive progress in evaluation metrics such as the pixel-wise relative error, most methods neglect the geometric constraints in the 3D space. In this work, we show the importance of the high-order 3D geometric constraints for depth prediction. By designing a loss term that enforces one simple type of geometric constraints, namely, virtual normal directions determined by randomly sampled three points in the reconstructed 3D space, we can considerably improve the depth pre…
Insufficient labeled training datasets is one of the bottlenecks of 3D hand pose estimation from monocular RGB images. Synthetic datasets have a large number of images with precise annotations, but the obvious difference with real-world datasets impacts the generalization. Little work has been done to bridge the gap between two domains over their wide difference. In this paper, we propose a domain adaptation method called Adaptive Wasserstein Hourglass (AW Hourglass) for weakly-supervised 3D hand pose estimation, which aims to distinguish the difference and explore the common characteristics (…
The Australian National University（澳大利亚国立大学）；Monash University（莫纳什大学）；Data61-CSIRO（Data61-澳大利亚联邦科学与工业研究组织）
Neural networks notoriously suffer from the problem of catastrophic forgetting, the phenomenon of forgetting the past knowledge when acquiring new knowledge. Overcoming catastrophic forgetting is of significant importance to emulate the process of "incremental learning", where the model is capable of learning from sequential experience in an efficient and robust way. State-of-the-art techniques for incremental learning make use of knowledge distillation towards preventing catastrophic forgetting. Therein, one updates the network while ensuring that the network's responses to previously seen co…
UMass Amherst（马萨诸塞大学阿默斯特分校）；University of Chicago（芝加哥大学）；TTI-Chicago（芝加哥丰田技术研究所）
As an agent moves through the world, the apparent motion of scene elements is (usually) inversely proportional to their depth. It is natural for a learning agent to associate image patterns with the magnitude of their displacement over time: as the agent moves, faraway mountains don't move much; nearby trees move a lot. This natural relationship between the appearance of objects and their motion is a rich source of information about the world. In this work, we start by training a deep network, using fully automatic supervision, to predict relative scene depth from single images. The relative d…
VicarVision（VicarVision公司）；Vilniaus Gedimino Technikos Universitetas（维尔纽斯格迪米纳斯技术大学）
This paper explores the capabilities of convolutional neural networks to deal with a task that is easily manageable for humans: perceiving 3D pose of a human body from varying angles. However, in our approach, we are restricted to using a monocular vision system. For this purpose, we apply a convolutional neural network approach on RGB videos and extend it to three dimensional convolutions. This is done via encoding the time dimension in videos as the 3\ts{rd} dimension in convolutional space, and directly regressing to human body joint positions in 3D coordinate space. This research shows the…
Artificial Intelligence Research Institute, Federal Research Center “Computer Science and Control” of Russian Academy of Sciences（俄罗斯科学院联邦研究中心“计算机科学与控制”人工智能研究所）；Peoples’ Friendship University of Russia (RUDN University)（俄罗斯人民友谊大学（RUDN大学））；Moscow Institute of Physics and Technology（莫斯科物理技术学院）
Vision-based depth reconstruction is a challenging problem extensively studied in computer vision but still lacking universal solution. Reconstructing depth from single image is particularly valuable to mobile robotics as it can be embedded to the modern vision-based simultaneous localization and mapping (vSLAM) methods providing them with the metric information needed to construct accurate maps in real scale. Typically, depth reconstruction is done nowadays via fully-convolutional neural networks (FCNNs). In this work we experiment with several FCNN architectures and introduce a few enhanceme…
