Localizing stereo boundaries is difficult because matching cues are absent in the occluded regions that are adjacent to them. We introduce an energy and level-set optimizer that improves boundaries by encoding the essential geometry of occlusions: The spatial extent of an occlusion must equal the amplitude of the disparity jump that causes it. In a collection of figure-ground scenes from Middlebury and Falling Things stereo datasets, the model provides more accurate boundaries than previous occlusion-handling techniques.
Recently, there are emerging many stereo matching methods for autonomous driving based on unsupervised learning. Most of them take advantage of reconstruction losses to remove dependency on disparity groundtruth. Occlusion handling is a challenging problem in stereo matching, especially for unsupervised methods. Previous unsupervised methods failed to take full advantage of geometry properties in occlusion handling. In this paper, we introduce an effective way to detect occlusion regions and propose a novel unsupervised training strategy to deal with occlusion that only uses the predicted left…
In this paper, we contrive a stereo matching algorithm with careful handling of disparity, discontinuity and occlusion. This algorithm works a worldwide matching stereo model which is based on minimization of energy. The global energy comprises two terms, firstly the data term and secondly the smoothness term. The data term is approximated by a color-weighted correlation, then refined in obstruct and low-texture areas in many applications of hierarchical loopy belief propagation algorithm. The results during the experiment are evaluated on the Middlebury data sets, showing that out algorithm i…
Pseudo-Stereo Inputs: A Solution to the Occlusion Challenge in Self-Supervised Stereo Matching
Shenyang Institute of Automation, Chinese Academy of Sciences（沈阳自动化研究所，中国科学院）；Liaoning Liaohe Laboratory（辽宁辽河实验室）；University of Chinese Academy of Sciences（中国科学院大学）；Key Laboratory on Intelligent Detection and Equipment Technology of Liaoning Province（辽宁省智能检测与装备技术重点实验室）
Localizing stereo boundaries and predicting nearby disparities are difficult because stereo boundaries induce occluded regions where matching cues are absent. Most modern computer vision algorithms treat occlusions secondarily (e.g., via left-right consistency checks after matching) or rely on high-level cues to improve nearby disparities (e.g., via deep networks and large training sets). They ignore the geometry of stereo occlusions, which dictates that the spatial extent of occlusion must equal the amplitude of the disparity jump that causes it. This paper introduces an energy and level-set …
In this paper, we extend the standard belief propagation (BP) sequential technique proposed in the tree-reweighted sequential method to the fully connected CRF models with the geodesic distance affinity. The proposed method has been applied to the stereo matching problem. Also a new approach to the BP marginal solution is proposed that we call one-view occlusion detection (OVOD). In contrast to the standard winner takes all (WTA) estimation, the proposed OVOD solution allows to find occluded regions in the disparity map and simultaneously improve the matching result. As a result we can perform…
Detecting the occlusion from stereo images or video frames is important to many computer vision applications. Previous efforts focus on bundling it with the computation of disparity or optical flow, leading to a chicken-and-egg problem. In this paper, we leverage convolutional neural network to liberate the occlusion detection task from the interleaved, traditional calculation framework. We propose a Symmetric Network (SymmNet) to directly exploit information from an image pair, without estimating disparity or motion in advance. The proposed network is structurally left-right symmetric to lear…
Robotics Institute, Carnegie Mellon University（卡内基梅隆大学机器人研究所）
Stereo reconstruction models trained on small images do not generalize well to high-resolution data. Training a model on high-resolution image size faces difficulties of data availability and is often infeasible due to limited computing resources. In this work, we present the Occlusion-aware Recurrent binocular Stereo matching (ORStereo), which deals with these issues by only training on available low disparity range stereo images. ORStereo generalizes to unseen high-resolution images with large disparity ranges by formulating the task as residual updates and refinements of an initial predicti…
SED-MVS: Segmentation-Driven and Edge-Aligned Deformation Multi-View Stereo with Depth Restoration and Occlusion Constraint
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；School of Electrical Engineering and Computer Science, The University of Queensland（昆士兰大学电气工程与计算机科学学院）；Hunan Police Academy（湖南警察学院）；Cooperative MediaNet Innovation Center, Shanghai Jiao Tong University（上海交通大学协同媒体网络创新中心）；School of Information Science & Engineering, Lanzhou University（兰州大学信息科学与工程学院）；Agricultural Information Institute, Chinese Academy of Agricultural Sciences（中国农业科学院农业信息研究所）
Toyota Technological Institute at Chicago（芝加哥丰田技术研究院）
In this paper we present a novel slanted-plane MRF model which reasons jointly about occlusion boundaries as well as depth. We formulate the problem as the one of inference in a hybrid MRF composed of both continuous (i.e., slanted 3D planes) and discrete (i.e., occlusion boundaries) random variables. This allows us to define potentials encoding the ownership of the pixels that compose the boundary between segments, as well as potentials encoding which junctions are physically possible. Our approach outperforms the state-of-the-art on Middlebury high resolution imagery as well as in the more c…
Stereo matching is a key component of autonomous driving perception. Recent unsupervised stereo matching approaches have received adequate attention due to their advantage of not requiring disparity ground truth. These approaches, however, perform poorly near occlusions. To overcome this drawback, in this paper, we propose CoT-Stereo, a novel unsupervised stereo matching approach. Specifically, we adopt a co-teaching framework where two networks interactively teach each other about the occlusions in an unsupervised fashion, which greatly improves the robustness of unsupervised stereo matching.…
German Research Center for Artificial Intelligence - DFKI（德国人工智能研究中心（DFKI））；Birla Institute of Technology and Science - BITS Pilani（贝拉理工学院（BITS Pilani））；University of Kaiserslautern - TUK（凯泽斯劳滕大学（TUK））
In the last few years, convolutional neural networks (CNNs) have demonstrated increasing success at learning many computer vision tasks including dense estimation problems such as optical flow and stereo matching. However, the joint prediction of these tasks, called scene flow, has traditionally been tackled using slow classical methods based on primitive assumptions which fail to generalize. The work presented in this paper overcomes these drawbacks efficiently (in terms of speed and accuracy) by proposing PWOC-3D, a compact CNN architecture to predict scene flow from stereo image sequences i…
Back to the Future Cyclopean Stereo: a human perception approach combining deep and geometric constraints
Instituto de Ciências Matemáticas e de Computação, Universidade de São Paulo (ICMC-USP)（圣保罗大学数学与计算机科学研究所（ICMC-USP））；Courant Institute of Mathematical Sciences, New York University (NYU)（纽约大学柯朗数学科学研究所（NYU））；Instituto de Matemática Pura e Aplicada (IMPA)（纯数学与应用数学研究所（IMPA））
In this paper we present ActiveStereoNet, the first deep learning solution for active stereo systems. Due to the lack of ground truth, our method is fully self-supervised, yet it produces precise depth with a subpixel precision of $1/30th$ of a pixel; it does not suffer from the common over-smoothing issues; it preserves the edges; and it explicitly handles occlusions. We introduce a novel reconstruction loss that is more robust to noise and texture-less patches, and is invariant to illumination changes. The proposed loss is optimized using a window-based cost aggregation with an adaptive supp…
In this paper, we present a decomposition model for stereo matching to solve the problem of excessive growth in computational cost (time and memory cost) as the resolution increases. In order to reduce the huge cost of stereo matching at the original resolution, our model only runs dense matching at a very low resolution and uses sparse matching at different higher resolutions to recover the disparity of lost details scale-by-scale. After the decomposition of stereo matching, our model iteratively fuses the sparse and dense disparity maps from adjacent scales with an occlusion-aware mask. A re…
Stereo Anywhere: Robust Zero-Shot Deep Stereo Matching Even Where Either Stereo or Mono Fail
Advanced Research Center on Electronic System (ARCES)（电子系统高级研究中心）；Department of Computer Science and Engineering (DISI)（计算机科学与工程系）；University of Bologna（博洛尼亚大学）
In this paper, we present confidence inference approachin an unsupervised way in stereo matching. Deep Neu-ral Networks (DNNs) have recently been achieving state-of-the-art performance. However, it is often hard to tellwhether the trained model was making sensible predictionsor just guessing at random. To address this problem, westart from a probabilistic interpretation of theL1loss usedin stereo matching, which inherently assumes an indepen-dent and identical (aka i.i.d.) Laplacian distribution. Weshow that with the newly introduced dense confidence map,the identical assumption is relaxed. In…
University of Michigan（密歇根大学）；TuSimple, Inc.（图森未来公司）
Dense stereo matching with deep neural networks is of great interest to the research community. Existing stereo matching networks typically use slow and computationally expensive 3D convolutions to improve the performance, which is not friendly to real-world applications such as autonomous driving. In this paper, we propose the Efficient Stereo Network (ESNet), which achieves high performance and efficient inference at the same time. ESNet relies only on 2D convolution and computes multi-scale cost volume efficiently using a warping-based method to improve the performance in regions with fine-…
Intelligent Media Institute, Communication University of Zhejiang（浙江传媒学院智能媒体研究院）；Key Lab of Film and TV Media Technology of Zhejiang Province（浙江省影视媒体技术重点实验室）；Software Engineering School, Zhejiang University（浙江大学软件学院）
Mixed reality applications often require virtual objects that are partly occluded by real objects. However, previous research and commercial products have limitations in terms of performance and efficiency. To address these challenges, we propose a novel depth contour occlusion (DCO) algorithm. The proposed method is based on the sensitivity of contour occlusion and a binocular stereoscopic vision device. In this method, a depth contour map is combined with a sparse depth map obtained from a two-stage adaptive filter area stereo matching algorithm and the depth contour information of the objec…
SenseTime Research（商汤科技研究院）；The Chinese University of Hong Kong（香港中文大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Shanghai AI Laboratory（上海人工智能实验室）；Qing Yuan Research Institute, Shanghai Jiao Tong University（上海交通大学清源研究院）
Recently, records on stereo matching benchmarks are constantly broken by end-to-end disparity networks. However, the domain adaptation ability of these deep models is quite poor. Addressing such problem, we present a novel domain-adaptive pipeline called AdaStereo that aims to align multi-level representations for deep stereo matching networks. Compared to previous methods for adaptive stereo matching, our AdaStereo realizes a more standard, complete and effective domain adaptation pipeline. Firstly, we propose a non-adversarial progressive color transfer algorithm for input image-level alignm…
Northwestern Polytechnical University（西北工业大学）；Huazhong University of Science and Technology（华中科技大学）
Seeking consistent point-to-point correspondences between 3D rigid data (point clouds, meshes, or depth maps) is a fundamental problem in 3D computer vision. While a number of correspondence selection methods have been proposed in recent years, their advantages and shortcomings remain unclear regarding different applications and perturbations. To fill this gap, this paper gives a comprehensive evaluation of nine state-of-the-art 3D correspondence grouping methods. A good correspondence grouping algorithm is expected to retrieve as many as inliers from initial feature matches, giving a rise in …
Learning Intra-view and Cross-view Geometric Knowledge for Stereo Matching
School of Electrical and Electronic Engineering, Nanyang Technological University（南洋理工大学电气与电子工程学院）；Institute for Infocomm Research, A*STAR（新加坡科技研究局信息通信研究院）
The performance of PatchMatch-based multi-view stereo algorithms depends heavily on the source views selected for computing matching costs. Instead of modeling the visibility of different views, most existing approaches handle occlusions in an ad-hoc manner. To address this issue, we propose a novel visibility-guided pixelwise view selection scheme in this paper. It progressively refines the set of source views to be used for each pixel in the reference view based on visibility information provided by already validated solutions. In addition, the Artificial Multi-Bee Colony (AMBC) algorithm is…
Tissue deformation during the surgery may significantly decrease the accuracy of surgical navigation systems. In this paper, we propose an approach to estimate the deformation of tissue surface from stereo videos in real-time, which is capable of handling occlusion, smooth surface and fast deformation. We first use a stereo matching method to extract depth information from stereo video frames and generate the tissue template, and then estimate the deformation of the obtained template by minimizing ICP, ORB feature matching and as-rigid-as-possible (ARAP) costs. The main novelties are twofold: …
CogStereo: Neural Stereo Matching with Implicit Spatial Cognition Embedding
Robotics and Computer Vision (RCV) Laboratory, Southern University of Science and Technology (SUSTech)（机器人与计算机视觉实验室，南方科技大学）；International Digital Economy Academy (IDEA)（国际数字经济学院）；School of Automation Science and Engineering, Xi’an Jiaotong University (XJTU)（自动化科学与工程学院，西安交通大学）
DMS:Diffusion-Based Multi-Baseline Stereo Generation for Improving Self-Supervised Depth Estimation
Institute of Science Tokyo, Japan（东京科学研究所）；Sony Semiconductor Solutions Group, Japan（日本索尼半导体解决方案集团）；Nanyang Technological University, Singapore（南洋理工大学）
Deep networks for stereo matching typically leverage 2D or 3D convolutional encoder-decoder architectures to aggregate cost and regularize the cost volume for accurate disparity estimation. Due to content-insensitive convolutions and down-sampling and up-sampling operations, these cost aggregation mechanisms do not take full advantage of the information available in the images. Disparity maps suffer from over-smoothing near occlusion boundaries, and erroneous predictions in thin structures. In this paper, we show how deep adaptive filtering and differentiable semi-global aggregation can be int…
SenseTime Group Limited（商汤科技集团）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Chinese University of Hong Kong（香港中文大学）；ShanghaiTech University（上海科技大学）
Recently, records on stereo matching benchmarks are constantly broken by end-to-end disparity networks. However, the domain adaptation ability of these deep models is quite limited. Addressing such problem, we present a novel domain-adaptive approach called AdaStereo that aims to align multi-level representations for deep stereo matching networks. Compared to previous methods, our AdaStereo realizes a more standard, complete and effective domain adaptation pipeline. Firstly, we propose a non-adversarial progressive color transfer algorithm for input image-level alignment. Secondly, we design a…
Self-Assessed Generation: Trustworthy Label Generation for Optical Flow and Stereo Matching in Real-world
School of Nanjing University of Science and Technology（南京理工大学）；School of Southeast University（东南大学）；Center for Frontier AI Research, Agency for Science, Technology and Research（新加坡科技研究局前沿人工智能研究中心）；Nanjing University of Information Science and Technology（南京信息工程大学）
Generalization Evaluation of Deep Stereo Matching Methods for UAV-Based Forestry Applications
Centre for Data Science and Artificial Intelligence（数据科学与人工智能中心）；Victoria University of Wellington（惠灵顿维多利亚大学）；Department of Computer Science and Software Engineering（计算机科学与软件工程系）；University of Canterbury（坎特伯雷大学）
A Lightweight Target-Driven Network of Stereo Matching for Inland Waterways
College of Artificial Intelligence, Tianjin University of Science and Technology（天津科技大学人工智能学院）；School of Artificial Intelligence, Anhui University（安徽大学人工智能学院）；Tianjin Institute of Navigation Instruments（天津航海仪器研究所）
Diving into the Fusion of Monocular Priors for Generalized Stereo Matching
Beijing Key Laboratory of Intelligent Information Technology, School of Computer Science & Technology, Beijing Institute of Technology, China（北京智能信息科技重点实验室，计算机科学与技术学院，北京理工大学，中国）；Guangdong Laboratory of Machine Perception and Intelligent Computing, Shenzhen MSU-BIT University, China（广东机器感知与智能计算实验室，深圳MSU-BIT大学，中国）；NVIDIA
Geometry Reinforced Efficient Attention Tuning Equipped with Normals for Robust Stereo Matching
Department of Computer Science, City University of Hong Kong（香港城市大学计算机科学系）；Department of Computer Science, Southern Methodist University（南卫理公会大学计算机科学系）；Hon Hai Research Institute（鸿海研究院）
IGEV++: Iterative Multi-range Geometry Encoding Volumes for Stereo Matching
School of Electronic Information and Communications, Huazhong University of Science and Technology（电子信息与通信学院，华中科技大学）；HiScene Information Technology Co., Ltd.（HiScene信息技术有限公司）
Crowded Video Individual Counting Informed by Social Grouping and Spatial-Temporal Displacement Priors
State Key Laboratory of Multispectral Information Intelligent Processing Technology（多谱信息智能处理技术国家重点实验室；人工智能与自动化学院，华中科技大学）；School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（湖北省智能机器人重点实验室；计算机科学与工程人工智能学院，武汉理工大学）；Hubei Key Laboratory of Intelligent Robot（软件工程学院，华中科技大学）；School of Computer Science & Engineering Artificial Intelligence, Wuhan Institute of Technology；School of Software Engineering, Huazhong University of Science and Technology
