Rethinking the Key Factors for the Generalization of Remote Sensing Stereo Matching Networks
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences（中国科学院大学电子电气与通信工程学院）
Stereo matching, a critical step of 3D reconstruction, has fully shifted towards deep learning due to its strong feature representation of remote sensing images. However, ground truth for stereo matching task relies on expensive airborne LiDAR data, thus making it difficult to obtain enough samples for supervised learning. To improve the generalization ability of stereo matching networks on cross-domain data from different sensors and scenarios, in this paper, we dedicate to study key training factors from three perspectives. (1) For the selection of training dataset, it is important to select data with similar regional target distribution as the test set instead of utilizing data from the same sensor. (2) For model structure, cascaded structure that flexibly adapts to different sizes of features is preferred. (3) For training manner, unsupervised methods generalize better than supervised methods, and we design an unsupervised early-stop strategy to help retain the best model with pre-trained weights as the basis. Extensive experiments are conducted to support the previous findings, on the basis of which we present an unsupervised stereo matching network with good generalization performance. We release the source code and the datasets at https://github.com/Elenairene/RKF_RSSM to reproduce the results and encourage future work.
GigaMVS presents several challenges to existing Multi-View Stereo (MVS) algorithms for its large scale, complex occlusions, and gigapixel images. To address these problems, we first apply one of the state-of-the-art learning-based MVS methods, --MVSFormer, to overcome intractable scenarios such as textureless and reflections regions suffered by traditional PatchMatch methods, but it fails in a few large scenes' reconstructions. Moreover, traditional PatchMatch algorithms such as ACMMP, OpenMVS, and RealityCapture are leveraged to further improve the completeness in large scenes. Furthermore, t…
Rethinking Monocular Depth Embedding for Generalized Stereo Matching
Shaanxi Key Laboratory for Network Computing and Security Technology, Xi’an University of Technology（陕西理工大学网络计算与安全技术重点实验室）；College of Computer Science, Sichuan University（四川大学计算机学院）
Depth estimation is solved as a regression or classification problem in existing learning-based multi-view stereo methods. Although these two representations have recently demonstrated their excellent performance, they still have apparent shortcomings, e.g., regression methods tend to overfit due to the indirect learning cost volume, and classification methods cannot directly infer the exact depth due to its discrete prediction. In this paper, we propose a novel representation, termed Unification, to unify the advantages of regression and classification. It can directly constrain the cost volu…
SDL-MVS: View Space and Depth Deformable Learning Paradigm for Multi-View Stereo Reconstruction in Remote Sensing
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；University of Chinese Academy of Sciences（中国科学院大学）；School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences（中国科学院大学电子电气与通信工程学院）
From Transparent to Opaque: Rethinking Neural Implicit Surfaces with $α$-NeuS
Key Laboratory of System Software (CAS) and State Key Laboratory of Computer Science, Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）；University of Chinese Academy of Sciences（中国科学院大学）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）；Department of Computer Science, Stony Brook University（石溪大学计算机科学系）；SenseTime Research（商汤科技研究院）
Advancing Applications of Satellite Photogrammetry: Novel Approaches for Built-up Area Modeling and Natural Environment Monitoring using Stereo/Multi-view Satellite Image-derived 3D Data
UC San Diego（加州大学圣地亚哥分校）；Cyprus International University（塞浦路斯国际大学）；the Hong Kong University of Science and Technology（香港科技大学）；University of Thessaloniki（塞萨洛尼基大学）
Potholes are one of the most common forms of road damage, which can severely affect driving comfort, road safety and vehicle condition. Pothole detection is typically performed by either structural engineers or certified inspectors. This task is, however, not only hazardous for the personnel but also extremely time-consuming. This paper presents an efficient pothole detection algorithm based on road disparity map estimation and segmentation. We first generalize the perspective transformation by incorporating the stereo rig roll angle. The road disparities are then estimated using semi-global m…
Wuhan University（武汉大学）；The Hong Kong University of Science and Technology（香港科技大学）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；XGRIDS；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
Existing learning-based multi-view stereo (MVS) methods rely on the depth range to build the 3D cost volume and may fail when the range is too large or unreliable. To address this problem, we propose a disparity-based MVS method based on the epipolar disparity flow (E-flow), called DispMVS, which infers the depth information from the pixel movement between two views. The core of DispMVS is to construct a 2D cost volume on the image plane along the epipolar line between each pair (between the reference image and several source images) for pixel matching and fuse uncountable depths triangulated …
Stereo matching has recently witnessed remarkable progress using Deep Neural Networks (DNNs). But, how robust are they? Although it has been well-known that DNNs often suffer from adversarial vulnerability with a catastrophic drop in performance, the situation is even worse in stereo matching. This paper first shows that a type of weak white-box attacks can overwhelm state-of-the-art methods. The attack is learned by a proposed stereo-constrained projected gradient descent (PGD) method in stereo matching. This observation raises serious concerns for the deployment of DNN-based stereo matching.…
3D recovery from multi-stereo and stereo images, as an important application of the image-based perspective geometry, serves many applications in computer vision, remote sensing and Geomatics. In this chapter, the authors utilize the imaging geometry and present approaches that perform 3D reconstruction from cross-view images that are drastically different in their viewpoints. We introduce our framework that takes ground-view images and satellite images for full 3D recovery, which includes necessary methods in satellite and ground-based point cloud generation from images, 3D data co-registrati…
Diachronic Stereo Matching for Multi-Date Satellite Imagery
IIE, Facultad de Ingeniería, Universidad de la República（乌拉圭国家工程院，工程学院，乌拉圭共和国大学）；Politecnico di Torino（托斯卡纳理工学院）；Eurecat, Centre Tecnologic de Catalunya（欧洲催化中心，加泰罗尼亚技术中心）；AMIAD, Pôle Recherche（AMIAD研究极）；Universite Paris-Saclay, CNRS, ENS Paris-Saclay, Centre Borelli（巴黎萨克雷大学，国家科学研究中心，巴黎萨克雷高等师范学校，博雷利中心）；Institut Universitaire de France（法国国家科学研究中心）
Deep learning (DL) stereo matching methods gained great attention in remote sensing satellite datasets. However, most of these existing studies conclude assessments based only on a few/single stereo images lacking a systematic evaluation on how robust DL methods are on satellite stereo images with varying radiometric and geometric configurations. This paper provides an evaluation of four DL stereo matching methods through hundreds of multi-date multi-site satellite stereo pairs with varying geometric configurations, against the traditional well-practiced Census-SGM (Semi-global matching), to c…
RSGround-R1: Rethinking Remote Sensing Visual Grounding through Spatial Reasoning
School of Electrical and Electronic Engineering, Nanyang Technological University（电气电子工程学院，南洋理工大学）；MoE Key Laboratory of Interdisciplinary Research of Computation and Economics, Shanghai University of Finance and Economics（教育部计算与经济交叉学科重点实验室，上海财经大学）
Why Do VLMs Fail at Remote Physiological Measurement? Rethinking with Physiological Grounding and Numerical Alignment
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Sichuan University（四川大学）；Nanyang Technological University（南洋理工大学）；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）
Remote Sensing Technology Institute, German Aerospace Center (DLR)（德国航空航天中心（DLR）遥感技术研究所）；Technical University of Munich（慕尼黑工业大学）
We describe the workflow of a digital surface models (DSMs) refinement algorithm using a hybrid conditional generative adversarial network (cGAN) where the generative part consists of two parallel networks merged at the last stage forming a WNet architecture. The inputs to the so-called WNet-cGAN are stereo DSMs and panchromatic (PAN) half-meter resolution satellite images. Fusing these helps to propagate fine detailed information from a spectral image and complete the missing 3D knowledge from a stereo DSM about building shapes. Besides, it refines the building outlines and edges making them …
Trust Before Autonomy: Rethinking Agentic Remote Sensing for Crisis Response
Harokopio University of Athens（雅典哈罗科皮奥大学）；Netcompany-Intrasoft S.A.（Netcompany-Intrasoft公司）；Dimocritus University of Thrace（色雷斯德谟克利特大学）；University of Western Macedonia（西马其顿大学）
The University of Sydney（悉尼大学）；SenseTime Computer Vision Research Group（商汤科技计算机视觉研究组）；SenseTime Research（商汤科技研究院）；Dalian University of Technology（大连理工大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
The recently proposed pseudo-LiDAR based 3D detectors greatly improve the benchmark of monocular/stereo 3D detection task. However, the underlying mechanism remains obscure to the research community. In this paper, we perform an in-depth investigation and observe that the efficacy of pseudo-LiDAR representation comes from the coordinate transformation, instead of data representation itself. Based on this observation, we design an image based CNN detector named Patch-Net, which is more generalized and can be instantiated as pseudo-LiDAR based 3D detectors. Moreover, the pseudo-LiDAR data in our…
Microsoft Research Asia（微软亚洲研究院）；Nanjing University（南京大学）；University of Science and Technology（中国科学技术大学）
The VirtualCube system is a 3D video conference system that attempts to overcome some limitations of conventional technologies. The key ingredient is VirtualCube, an abstract representation of a real-world cubicle instrumented with RGBD cameras for capturing the 3D geometry and texture of a user. We design VirtualCube so that the task of data capturing is standardized and significantly simplified, and everything can be built using off-the-shelf hardware. We use VirtualCubes as the basic building blocks of a virtual conferencing environment, and we provide each VirtualCube user with a surroundi…
GeoChrono: Benchmarking and Rethinking Long-Term Temporal Understanding in Remote Sensing
Beijing University of Posts and Telecommunications（北京邮电大学）；Tsinghua University（清华大学）；Hunan Normal University（湖南师范大学）；City University of Hong Kong（香港城市大学）
Deep learning (DL) methods are widely investigated for stereo image matching tasks due to their reported high accuracies. However, their transferability/generalization capabilities are limited by the instances seen in the training data. With satellite images covering large-scale areas with variances in locations, content, land covers, and spatial patterns, we expect their performances to be impacted. Increasing the number and diversity of training data is always an option, but with the ground-truth disparity being limited in remote sensing due to its high cost, it is almost impossible to obtai…
Opening the Black Box of 3D Reconstruction Error Analysis with VECTOR
Northeastern University（东北大学）；Wētā FX（维塔数码）；Art Center College of Design（艺术中心设计学院）；Jet Propulsion Laboratory（喷气推进实验室）；California Institute of Technology（加州理工学院）；Carnegie Mellon University（卡内基梅隆大学）
Beijing Institute of Technology（北京理工大学）；German Aerospace Center (DLR)（德国航空航天中心（DLR））；Mississippi State University（密西西比州立大学）
Vehicle detection in remote sensing images has attracted increasing interest in recent years. However, its detection ability is limited due to lack of well-annotated samples, especially in densely crowded scenes. Furthermore, since a list of remotely sensed data sources is available, efficient exploitation of useful information from multi-source data for better vehicle detection is challenging. To solve the above issues, a multi-source active fine-tuning vehicle detection (Ms-AFt) framework is proposed, which integrates transfer learning, segmentation, and active classification into a unified …
Rochester Institute of Technology（罗切斯特理工学院）；Xerra Earth Observation Institute（Xerra地球观测研究所）
When 3D-point clouds from overhead sensors are used as input to remote sensing data exploitation pipelines, a large amount of effort is devoted to data preparation. Among the multiple stages of the preprocessing chain, estimating the Digital Terrain Model (DTM) model is considered to be of a high importance; however, this remains a challenge, especially for raw point clouds derived from optical imagery. Current algorithms estimate the ground points using either a set of geometrical rules that require tuning multiple parameters and human interaction, or cast the problem as a binary classificati…
