This work introduces an effective and practical solution to the dense two-view structure from motion (SfM) problem. One vital question addressed is how to mindfully use per-pixel optical flow correspondence between two frames for accurate pose estimation -- as perfect per-pixel correspondence between two images is difficult, if not impossible, to establish. With the carefully estimated camera pose and predicted per-pixel optical flow correspondences, a dense depth of the scene is computed. Later, an iterative refinement procedure is introduced to further improve optical flow matching confidence, camera pose, and depth, exploiting their inherent dependency in rigid SfM. The fundamental idea presented is to benefit from per-pixel uncertainty in the optical flow estimation and provide robustness to the dense SfM system via an online refinement. Concretely, we introduce our uncertainty-driven Dense Two-View SfM pipeline (DTV-SfM), consisting of an uncertainty-aware dense optical flow estimation approach that provides per-pixel correspondence with their confidence score of matching; a weighted dense bundle adjustment formulation that depends on optical flow uncertainty and bidirectional optical flow consistency to refine both pose and depth; a depth estimation network that considers its consistency with the estimated poses and optical flow respecting epipolar constraint. Extensive experiments show that the proposed approach achieves remarkable depth accuracy and state-of-the-art camera pose results superseding SuperPoint and SuperGlue accuracy when tested on benchmark datasets such as DeMoN, YFCC100M, and ScanNet. Code and more materials are available at http://vis.xyz/pub/dtv-sfm.
University of Missouri-Kansas City（密苏里大学堪萨斯城分校）；University of Texas at Arlington（得克萨斯大学阿灵顿分校）；Southern University of Science and Technology（南方科技大学）；National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所模式识别国家重点实验室）；Watrix technology limited co. ltd（Watrix科技有限公司）
Gait recognition has proven to be effective for long-distance human recognition. But view variance of gait features would change human appearance greatly and reduce its performance. Most existing gait datasets usually collect data with a dozen different angles, or even more few. Limited view angles would prevent learning better view invariant feature. It can further improve robustness of gait recognition if we collect data with various angles at 1 degree interval. But it is time consuming and labor consuming to collect this kind of dataset. In this paper, we, therefore, introduce a Dense-View …
This paper investigates conditional generative adversarial networks (cGANs) to overcome a fundamental limitation of using geotagged media for geographic discovery, namely its sparse and uneven spatial distribution. We train a cGAN to generate ground-level views of a location given overhead imagery. We show the "fake" ground-level images are natural looking and are structurally similar to the real images. More significantly, we show the generated images are representative of the locations and that the representations learned by the cGANs are informative. In particular, we show that dense featur…
Envision3D: One Image to 3D with Anchor Views Interpolation
School of Electronic and Computer Engineering, Peking University（北京大学电子与计算机工程学院）；National University of Singapore（新加坡国立大学）；PengCheng Laboratory（鹏城实验室）；Rabbitpre（睿比特科技）
University of Oxford（牛津大学）；University of Warwick（华威大学）
In this paper, we propose a novel approach, 3D-RecGAN++, which reconstructs the complete 3D structure of a given object from a single arbitrary depth view using generative adversarial networks. Unlike existing work which typically requires multiple views of the same object or class labels to recover the full 3D geometry, the proposed 3D-RecGAN++ only takes the voxel grid representation of a depth view of the object as input, and is able to generate the complete 3D occupancy grid with a high resolution of 256^3 by recovering the occluded/missing regions. The key idea is to combine the generativ…
Uncertainty for SVBRDF Acquisition using Frequency Analysis
ETH Zurich（苏黎世联邦理工学院）；Delft University of Technology（代尔夫特理工大学）；Adobe Research Switzerland（Adobe瑞士研究实验室）；Adobe Research USA（Adobe美国研究实验室）；Adobe Research UK（Adobe英国研究实验室）
University of Liverpool（利物浦大学）；University of Edinburgh（爱丁堡大学）
Dense prediction tasks are common for 3D point clouds, but the uncertainties inherent in massive points and their embeddings have long been ignored. In this work, we present CUE, a novel uncertainty estimation method for dense prediction tasks in 3D point clouds. Inspired by metric learning, the key idea of CUE is to explore cross-point embeddings upon a conventional 3D dense prediction pipeline. Specifically, CUE involves building a probabilistic embedding model and then enforcing metric alignments of massive points in the embedding space. We also propose CUE+, which enhances CUE by explicitl…
This project integrates infrared and RGB imagery to produce dense 3D environment models reconstructed from multiple views. The resulting 3D map contains both thermal and RGB information which can be used in robotic fire-fighting applications to identify victims and active fire areas.
University of Bern（伯尔尼大学）；University Hospital and University of Bern（伯尔尼大学医院和伯尔尼大学）
This paper proposes a novel approach for uncertainty quantification in dense Conditional Random Fields (CRFs). The presented approach, called Perturb-and-MPM, enables efficient, approximate sampling from dense multi-label CRFs via random perturbations. An analytic error analysis was performed which identified the main cause of approximation error as well as showed that the error is bounded. Spatial uncertainty maps can be derived from the Perturb-and-MPM model, which can be used to visualize uncertainty in image segmentation results. The method is validated on synthetic and clinical Magnetic R…
Australian National University（澳大利亚国立大学）；Northwestern Polytechnical University（西北工业大学）；Monash University（莫纳什大学）；SenseTime（商汤科技）
Uncertainty estimation has been extensively studied in recent literature, which can usually be classified as aleatoric uncertainty and epistemic uncertainty. In current aleatoric uncertainty estimation frameworks, it is often neglected that the aleatoric uncertainty is an inherent attribute of the data and can only be correctly estimated with an unbiased oracle model. Since the oracle model is inaccessible in most cases, we propose a new sampling and selection strategy at train time to approximate the oracle model for aleatoric uncertainty estimation. Further, we show a trivial solution in the…
Motivated by the need to identify erroneous disparity assignments, various approaches for uncertainty and confidence estimation of dense stereo matching have been presented in recent years. As in many other fields, especially deep learning based methods have shown convincing results. However, most of these methods only model the uncertainty contained in the data, while ignoring the uncertainty of the employed dense stereo matching procedure. Additionally modelling the latter, however, is particularly beneficial if the domain of the training data varies from that of the data to be processed. Fo…
DKPMV: Dense Keypoints Fusion from Multi-View RGB Frames for 6D Pose Estimation of Textureless Objects
College of Aerospace Science and Engineering, National University of Defense Technology（航空航天科学与工程学院，国防科技大学）；Hunan Provincial Key Laboratory of Image Measurement and Vision Navigation（图像测量与视觉导航省级重点实验室）
Active Implicit Object Reconstruction using Uncertainty-guided Next-Best-View Optimization
School of Mechanical Engineering and Automation, Harbin Institute of Technology Shenzhen（哈尔滨工业大学（深圳）机械工程与自动化学院）；Shenzhen University General Hospital（深圳大学总医院）
National University of Singapore（新加坡国立大学）；Yale-NUS College（耶鲁-新加坡国大学院）；Nanyang Technological University（南洋理工大学）
Few existing image defogging or dehazing methods consider dense and non-uniform particle distributions, which usually happen in smoke, dust and fog. Dealing with these dense and/or non-uniform distributions can be intractable, since fog's attenuation and airlight (or veiling effect) significantly weaken the background scene information in the input image. To address this problem, we introduce a structure-representation network with uncertainty feedback learning. Specifically, we extract the feature representations from a pre-trained Vision Transformer (DINO-ViT) module to recover the backgroun…
Technical University of Munich（慕尼黑工业大学）；Artisense；Karlsruhe University of Applied Sciences（卡尔斯鲁厄应用科学大学）
In this paper, we present TANDEM a real-time monocular tracking and dense mapping framework. For pose estimation, TANDEM performs photometric bundle adjustment based on a sliding window of keyframes. To increase the robustness, we propose a novel tracking front-end that performs dense direct image alignment using depth maps rendered from a global model that is built incrementally from dense depth predictions. To predict the dense depth maps, we propose Cascade View-Aggregation MVSNet (CVA-MVSNet) that utilizes the entire active keyframe window by hierarchically constructing 3D cost volumes wit…
We present a novel method to reconstruct 3D scenes from images by leveraging deep dense monocular SLAM and fast uncertainty propagation. The proposed approach is able to 3D reconstruct scenes densely, accurately, and in real-time while being robust to extremely noisy depth estimates coming from dense monocular SLAM. Differently from previous approaches, that either use ad-hoc depth filters, or that estimate the depth uncertainty from RGB-D cameras' sensor models, our probabilistic depth uncertainty derives directly from the information matrix of the underlying bundle adjustment problem in SLAM…
Technical University of Munich（慕尼黑工业大学）；University of Toulouse（图卢兹大学）；Mikros Image
We introduce a variational method for multi-view shape-from-shading under natural illumination. The key idea is to couple PDE-based solutions for single-image based shape-from-shading problems across multiple images and multiple color channels by means of a variational formulation. Rather than alternatingly solving the individual SFS problems and optimizing the consistency across images and channels which is known to lead to suboptimal results, we propose an efficient solution of the coupled problem by means of an ADMM algorithm. In numerous experiments on both simulated and real imagery, we d…
UGOD: Uncertainty-Guided Opacity and Dropout for Sparse-View 3D Gaussian Splatting
Manchester Metropolitan University（曼彻斯特城市大学）；University of Surrey（萨里大学）；Imperial College London（帝国理工学院）；University of Exeter（埃克塞特大学）；Southeast University（东南大学）；Nanjing University of Science and Technology（南京理工大学）；University of Leeds（利兹大学）；Aston University（阿斯顿大学）
MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）；Rutgers University（罗格斯大学）
Uncertainty estimation is an essential step in the evaluation of the robustness for deep learning models in computer vision, especially when applied in risk-sensitive areas. However, most state-of-the-art deep learning models either fail to obtain uncertainty estimation or need significant modification (e.g., formulating a proper Bayesian treatment) to obtain it. Most previous methods are not able to take an arbitrary model off the shelf and generate uncertainty estimation without retraining or redesigning it. To address this gap, we perform a systematic exploration into training-free uncertai…
Imperial College London（伦敦帝国理工学院）；Dyson Technology Ltd.（戴森科技有限公司）
While the keypoint-based maps created by sparse monocular simultaneous localisation and mapping (SLAM) systems are useful for camera tracking, dense 3D reconstructions may be desired for many robotic tasks. Solutions involving depth cameras are limited in range and to indoor spaces, and dense reconstruction systems based on minimising the photometric error between frames are typically poorly constrained and suffer from scale ambiguity. To address these issues, we propose a 3D reconstruction system that leverages the output of a convolutional neural network (CNN) to produce fully dense depth ma…
College of Intelligence and Computing, Tianjin University（天津大学智能与计算学部）；Tianjin Key Lab of Machine Learning（天津市机器学习重点实验室）
Learning from different data views by exploring the underlying complementary information among them can endow the representation with stronger expressive ability. However, high-dimensional features tend to contain noise, and furthermore, the quality of data usually varies for different samples (even for different views), i.e., one view may be informative for one sample but not the case for another. Therefore, it is quite challenging to integrate multi-view noisy data under unsupervised setting. Traditional multi-view methods either simply treat each view with equal importance or tune the weigh…
In this paper, we address the problem of estimating dense depth from a sequence of images using deep neural networks. Specifically, we employ a dense-optical-flow network to compute correspondences and then triangulate the point cloud to obtain an initial depth map.Parts of the point cloud, however, may be less accurate than others due to lack of common observations or small parallax. To further increase the triangulation accuracy, we introduce a depth-refinement network (DRN) that optimizes the initial depth map based on the image's contextual cues. In particular, the DRN contains an iterativ…
Thin, reflective objects such as forks and whisks are common in our daily lives, but they are particularly challenging for robot perception because it is hard to reconstruct them using commodity RGB-D cameras or multi-view stereo techniques. While traditional pipelines struggle with objects like these, Neural Radiance Fields (NeRFs) have recently been shown to be remarkably effective for performing view synthesis on objects with thin structures or reflective materials. In this paper we explore the use of NeRF as a new source of supervision for robust robot vision systems. In particular, we dem…
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Stellantis, Opel Automobile GmbH（斯特兰蒂斯，欧宝汽车有限公司）
At the heart of all automated driving systems is the ability to sense the surroundings, e.g., through semantic segmentation of LiDAR sequences, which experienced a remarkable progress due to the release of large datasets such as SemanticKITTI and nuScenes-LidarSeg. While most previous works focus on sparse segmentation of the LiDAR input, dense output masks provide self-driving cars with almost complete environment information. In this paper, we introduce MASS - a Multi-Attentional Semantic Segmentation model specifically built for dense top-view understanding of the driving scenes. Our framew…
School of Computing, the Australian National University（澳大利亚国立大学计算学院）；School of Electronics and Information, Northwestern Polytechnical University（西北工业大学电子信息学院）；CS, Nankai University（南开大学计算机学院）；CSIRO（澳大利亚联邦科学与工业研究组织）；Monash University（莫纳什大学）
Deep neural networks can be roughly divided into deterministic neural networks and stochastic neural networks.The former is usually trained to achieve a mapping from input space to output space via maximum likelihood estimation for the weights, which leads to deterministic predictions during testing. In this way, a specific weights set is estimated while ignoring any uncertainty that may occur in the proper weight space. The latter introduces randomness into the framework, either by assuming a prior distribution over model parameters (i.e. Bayesian Neural Networks) or including latent variable…
Uncertainty quantification is essential for robotic perception, as overconfident or point estimators can lead to collisions and damages to the environment and the robot. In this paper, we evaluate scalable approaches to uncertainty quantification in single-view supervised depth learning, specifically MC dropout and deep ensembles. For MC dropout, in particular, we explore the effect of the dropout at different levels in the architecture. We show that adding dropout in all layers of the encoder brings better results than other variations found in the literature. This configuration performs simi…
Hochschule Bonn-Rhein-Sieg（波恩-莱茵-锡格应用技术大学）；German Research Center for Artificial Intelligence（德国人工智能研究中心）
Application of underwater robots are on the rise, most of them are dependent on sonar for underwater vision, but the lack of strong perception capabilities limits them in this task. An important issue in sonar perception is matching image patches, which can enable other techniques like localization, change detection, and mapping. There is a rich literature for this problem in color images, but for acoustic images, it is lacking, due to the physics that produce these images. In this paper we improve on our previous results for this problem (Valdenegro-Toro et al, 2017), instead of modeling feat…
Hyper-V2X: Hypernetworks for Estimating Epistemic and Aleatoric Uncertainty in Cooperative Bird's-Eye-View Semantic Segmentation
CARISSMA Institute for Electric, COnnected, and Secure Mobility (C-ECOS), Technische Hochschule Ingolstadt（CARISSMA电动、连接与安全移动研究所（C-ECOS）、因戈尔施塔特技术大学）；University of Applied Sciences Aschaffenburg（阿施发堡应用科学大学）
In this paper, we tackle the problem of active robotic 3D reconstruction of an object. In particular, we study how a mobile robot with an arm-held camera can select a favorable number of views to recover an object's 3D shape efficiently. Contrary to the existing solution to this problem, we leverage the popular neural radiance fields-based object representation, which has recently shown impressive results for various computer vision tasks. However, it is not straightforward to directly reason about an object's explicit 3D geometric details using such a representation, making the next-best-view…
MU-GeNeRF: Multi-view Uncertainty-guided Generalizable Neural Radiance Fields for Distractor-aware Scene
School of Computer Science and Technology, Tongji University, Shanghai, China（同济大学计算机科学与技术学院，上海，中国）；MOE Key Lab of Embedded System and Service Computing, Tongji University, Shanghai, China（同济大学嵌入式系统与服务计算MOE重点实验室，上海，中国）；College of Surveying and Geographic Informatics, Tongji University, Shanghai, China（同济大学测绘地理信息学院，上海，中国）
From Sparse to Dense: Spatio-Temporal Fusion for Multi-View 3D Human Pose Estimation with DenseWarper
Department of Computer Science, THUAI, BNRist, Tsinghua University, Beijing, China（清华大学北京研究院，清华大学计算机科学系，北京，中国）；Dalian University of Technology, Dalian, China（大连理工大学，大连，中国）；Apple, Beijing, China（苹果公司，北京，中国）；Hong Kong University of Science and Technology (Guang Zhou), Guang Zhou, China（香港科技大学（广州），广州，中国）；University of Manchester, Manchester, UK（曼彻斯特大学，曼彻斯特，英国）
Technical University of Munich（慕尼黑工业大学）；Google Research（谷歌研究院）
Neural radiance fields (NeRF) encode a scene into a neural representation that enables photo-realistic rendering of novel views. However, a successful reconstruction from RGB images requires a large number of input views taken under static conditions - typically up to a few hundred images for room-size scenes. Our method aims to synthesize novel views of whole rooms from an order of magnitude fewer images. To this end, we leverage dense depth priors in order to constrain the NeRF optimization. First, we take advantage of the sparse depth data that is freely available from the structure from mo…
Learning dense visual representations without labels is an arduous task and more so from scene-centric data. We propose to tackle this challenging problem by proposing a Cross-view consistency objective with an Online Clustering mechanism (CrOC) to discover and segment the semantics of the views. In the absence of hand-crafted priors, the resulting method is more generalizable and does not require a cumbersome pre-processing step. More importantly, the clustering algorithm conjointly operates on the features of both views, thereby elegantly bypassing the issue of content not represented in bot…
Semi-Dense Matching Uncertainty Is Not Just Local Confidence
University of Engineering and Technology, Vietnam National University（越南国家大学工程技术大学）；Optoelectronics Center, Viettel Aerospace Institute, Viettel Group（Viettel集团Viettel宇航研究院光电中心）；National Yang Ming Chiao Tung University（国立阳明交通大学）
School of Electrical and Electronic Engineering, Nanyang Technological University (NTU), Singapore（南洋理工大学电子与电气工程学院）；Institute for Infocomm Research (I 2 R), Agency for Science, Technology and Research (A*STAR), Singapore（信息与通信研究 institute（I2R），科技研究局（A*STAR））；Institute of High Performance Computing, Agency for Science, Technology and Research (A*STAR), Singapore（高性能计算研究所，科技研究局（A*STAR））；College of Computer Science, Nankai University, Tianjin, China（南开大学计算机学院）
