People capture memorable images of events and exhibits that are often occluded by a wire mesh loosely termed as fence. Recent works in removing fence have limited performance due to the difficulty in initial fence segmentation. This work aims to accurately segment fence using a novel fence guidance mask (FM) generated from stereo image pair. This binary guidance mask contains deterministic cues about the structure of fence and is given as additional input to the deep fence estimation model. We also introduce a directional connectivity loss (DCL), which is used alongside adversarial loss to precisely detect thin wires. Experimental results obtained on real world scenarios demonstrate the superiority of proposed method over state-of-the-art techniques.
Indian Institute of Technology Kharagpur（印度理工学院卡哈拉格普尔分校）
Conventional approaches to image de-fencing have limited themselves to using only image data in adjacent frames of the captured video of an approximately static scene. In this work, we present a method to harness disparity using a stereo pair of fenced images in order to detect fence pixels. Tourists and amateur photographers commonly carry smartphones/phablets which can be used to capture a short video sequence of the fenced scene. We model the formation of the occluded frames in the captured video. Furthermore, we propose an optimization framework to estimate the de-fenced image using the to…
Aviation University of Air Force（空军航空大学）；JD Explore Academy, JD.COM（京东探索研究院，京东集团）；The University of Melbourne（墨尔本大学）
Currently, single image inpainting has achieved promising results based on deep convolutional neural networks. However, inpainting on stereo images with missing regions has not been explored thoroughly, which is also a significant but different problem. One crucial requirement for stereo image inpainting is stereo consistency. To achieve it, we propose an Iterative Geometry-Aware Cross Guidance Network (IGGNet). The IGGNet contains two key ingredients, i.e., a Geometry-Aware Attention (GAA) module and an Iterative Cross Guidance (ICG) strategy. The GAA module relies on the epipolar geometry cu…
Optimal Pose Guidance for Stereo Calibration in 3D Deformation Measurement
College of Aerospace Science and Engineering（航空航天科学与工程学院）；National University of Defense Technology（国防科技大学）；Hunan Provincial Key Laboratory of Image Measurement and Vision Navigation（湖南省图像测量与视觉导航重点实验室）；Hypersonic Technology Laboratory（高超声速技术实验室）；Hunan Key Laboratory of Intelligent Planning and Simulation for Aerospace Missions（湖南省航空航天任务智能规划与模拟重点实验室）
The performance of image based stereo estimation suffers from lighting variations, repetitive patterns and homogeneous appearance. Moreover, to achieve good performance, stereo supervision requires sufficient densely-labeled data, which are hard to obtain. In this work, we leverage small amount of data with very sparse but accurate disparity cues from LiDAR to bridge the gap. We propose a novel sparsity expansion technique to expand the sparse cues concerning RGB images for local feature enhancement. The feature enhancement method can be easily applied to any stereo estimation algorithms with …
This paper introduces a novel deep framework for dense 3D reconstruction from multiple image frames, leveraging a sparse set of depth measurements gathered jointly with image acquisition. Given a deep multi-view stereo network, our framework uses sparse depth hints to guide the neural network by modulating the plane-sweep cost volume built during the forward step, enabling us to infer constantly much more accurate depth maps. Moreover, since multiple viewpoints can provide additional depth measurements, we propose a multi-view guidance strategy that increases the density of the sparse points u…
Hefei University of Technology（合肥工业大学）；Orbbec（奥比中光）；Sun Yat-sen University（中山大学）
It is well known that the passive stereo system cannot adapt well to weak texture objects, e.g., white walls. However, these weak texture targets are very common in indoor environments. In this paper, we present a novel stereo system, which consists of two cameras (an RGB camera and an IR camera) and an IR speckle projector. The RGB camera is used both for depth estimation and texture acquisition. The IR camera and the speckle projector can form a monocular structured-light (MSL) subsystem, while the two cameras can form a binocular stereo subsystem. The depth map generated by the MSL subsyste…
SGDFormer: One-stage Transformer-based Architecture for Cross-Spectral Stereo Image Guided Denoising
College of Information Science and Electronic Engineering, Zhejiang University（浙江大学信息科学与工程学院）；Ningbo Research Institute, Zhejiang University（浙江大学宁波研究院）；Zhejiang GongShang University（浙江工商大学）
Recent studies of deep learning based stereo image super-resolution (StereoSR) have promoted the development of StereoSR. However, existing StereoSR models mainly concentrate on improving quantitative evaluation metrics and neglect the visual quality of super-resolved stereo images. To improve the perceptual performance, this paper proposes the first perception-oriented stereo image super-resolution approach by exploiting the feedback, provided by the evaluation on the perceptual quality of StereoSR results. To provide accurate guidance for the StereoSR model, we develop the first special ster…
National Key Laboratory of Science and Technology on Multispectral Information Processing（多光谱信息处理科学技术国家重点实验室）；School of Artifical Intelligence and Automation, Huazhong University of Science and Technology（华中科技大学人工智能与自动化学院）
In this paper, we propose an efficient multi-scale geometric consistency guided multi-view stereo method for accurate and complete depth map estimation. We first present our basic multi-view stereo method with Adaptive Checkerboard sampling and Multi-Hypothesis joint view selection (ACMH). It leverages structured region information to sample better candidate hypotheses for propagation and infer the aggregation view subset at each pixel. For the depth estimation of low-textured areas, we further propose to combine ACMH with multi-scale geometric consistency guidance (ACMM) to obtain the reliabl…
Beijing Institute of Technology（北京理工大学）；Kandao Australia Research Center（看到科技澳大利亚研究中心）
Deep neural networks have shown excellent performance for stereo matching. Many efforts focus on the feature extraction and similarity measurement of the matching cost computation step while less attention is paid on cost aggregation which is crucial for stereo matching. In this paper, we present a learning-based cost aggregation method for stereo matching by a novel sub-architecture in the end-to-end trainable pipeline. We reformulate the cost aggregation as a learning process of the generation and selection of cost aggregation proposals which indicate the possible cost aggregation results. T…
In this paper, we present a new perspective towards image-based shape generation. Most existing deep learning based shape reconstruction methods employ a single-view deterministic model which is sometimes insufficient to determine a single groundtruth shape because the back part is occluded. In this work, we first introduce a conditional generative network to model the uncertainty for single-view reconstruction. Then, we formulate the task of multi-view reconstruction as taking the intersection of the predicted shape spaces on each single image. We design new differentiable guidance including …
MonSter++: Unified Stereo Matching, Multi-view Stereo, and Real-time Stereo with Monodepth Priors
School of Electronic Information and Communications, Huazhong University of Science and Technology（电子信息与通讯学院，华中科技大学）；Meta；AI Chip Center, Hong Kong University of Science and Technology（香港科技大学人工智能芯片中心）；Nanjing Forestry University（南京林业大学）；Autel Robotics
GeoStereo: A Unified Stereo Geometry Estimation Framework for Disparity and Surface Normal
Beijing Institute of Technology（北京理工大学）；School of Computer Science, Wuhan University（武汉大学计算机科学学院）；Beihang University（北京航空航天大学）；INSAIT, Sofia University（索非亚大学INSAIT）；BAAI AIR, Tsinghua University（清华大学BAAI AIR）
Tsinghua University（清华大学）；University College London（伦敦大学学院）；Beijing Huajie IMI Technology Co., Ltd.（北京华捷艾米科技有限公司）
In human-computer interaction, it is important to accurately estimate the hand pose especially fingertips. However, traditional approaches for fingertip localization mainly rely on depth images and thus suffer considerably from the noise and missing values. Instead of depth images, stereo images can also provide 3D information of hands and promote 3D hand pose estimation. There are nevertheless limitations on the dataset size, global viewpoints, hand articulations and hand shapes in the publicly available stereo-based hand pose datasets. To mitigate these limitations and promote further resear…
Tracking Any Point Methods for Markerless 3D Tissue Tracking in Endoscopic Stereo Images
Institute of Medical Technology and Intelligent Systems, Hamburg University of Technology（医学技术与智能系统研究所，汉堡技术大学）；Department of Otorhinolaryngology, Head and Neck Surgery and Oncology, University Medical Center Hamburg-Eppendorf（耳鼻喉头颈外科与肿瘤学部，汉堡艾本多夫大学医学中心）；Martini-Klinik Prostate Cancer Center, University Medical Center Hamburg-Eppendorf（马尔廷诊所前列腺癌中心，汉堡艾本多夫大学医学中心）；Department of Urology, University Medical Center Hamburg-Eppendorf（泌尿外科，汉堡艾本多夫大学医学中心）
本文提出一种利用2D Tracking Any Point网络结合两个CoTracker模型实现内窥镜立体图像无标记3D组织追踪的新方法，实验验证了其在复杂手术场景中的准确性与潜力。
Depth estimation is an important capability for autonomous vehicles to understand and reconstruct 3D environments as well as avoid obstacles during the execution. Accurate depth sensors such as LiDARs are often heavy, expensive and can only provide sparse depth while lighter depth sensors such as stereo cameras are noiser in comparison. We propose an end-to-end learning algorithm that is capable of using sparse, noisy input depth for refinement and depth completion. Our model also produces the camera pose as a byproduct, making it a great solution for autonomous systems. We evaluate our approa…
The complementary fusion of light detection and ranging (LiDAR) data and image data is a promising but challenging task for generating high-precision and high-density point clouds. This study proposes an innovative LiDAR-guided stereo matching approach called LiDAR-guided stereo matching (LGSM), which considers the spatial consistency represented by continuous disparity or depth changes in the homogeneous region of an image. The LGSM first detects the homogeneous pixels of each LiDAR projection point based on their color or intensity similarity. Next, we propose a riverbed enhancement function…
Both Ears Wide Open: Towards Language-Driven Spatial Audio Generation
Hong Kong University of Science and Technology（香港科技大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Nanyang Technological University（南洋理工大学）；Zhejiang University（浙江大学）
University of Freiburg（弗莱堡大学）；University of Technology, Nuremberg（纽伦堡工业大学）
In this paper we propose USegScene, a framework for semantically guided unsupervised learning of depth, optical flow and ego-motion estimation for stereo camera images using convolutional neural networks. Our framework leverages semantic information for improved regularization of depth and optical flow maps, multimodal fusion and occlusion filling considering dynamic rigid object motions as independent SE(3) transformations. Furthermore, complementary to pure photo-metric matching, we propose matching of semantic features, pixel-wise classes and object instance borders between the consecutive …
