Geospatial object segmentation, as a particular semantic segmentation task, always faces with larger-scale variation, larger intra-class variance of background, and foreground-background imbalance in the high spatial resolution (HSR) remote sensing imagery. However, general semantic segmentation methods mainly focus on scale variation in the natural scene, with inadequate consideration of the other two problems that usually happen in the large area earth observation scene. In this paper, we argue that the problems lie on the lack of foreground modeling and propose a foreground-aware relation network (FarSeg) from the perspectives of relation-based and optimization-based foreground modeling, to alleviate the above two problems. From perspective of relation, FarSeg enhances the discrimination of foreground features via foreground-correlated contexts associated by learning foreground-scene relation. Meanwhile, from perspective of optimization, a foreground-aware optimization is proposed to focus on foreground examples and hard examples of background during training for a balanced optimization. The experimental results obtained using a large scale dataset suggest that the proposed method is superior to the state-of-the-art general semantic segmentation methods and achieves a better trade-off between speed and accuracy. Code has been made available at: \url{https://github.com/Z-Zheng/FarSeg}.
German Aerospace Center（德国航空航天中心）；IBM T.J. Watson Research Center（IBM T.J.沃森研究中心）
A key challenge of supervised learning is the availability of human-labeled data. We evaluate a big data processing pipeline to auto-generate labels for remote sensing data. It is based on rasterized statistical features extracted from surveys such as e.g. LiDAR measurements. Using simple combinations of the rasterized statistical layers, it is demonstrated that multiple classes can be generated at accuracies of ~0.9. As proof of concept, we utilize the big geo-data platform IBM PAIRS to dynamically generate such labels in dense urban areas with multiple land cover classes. The general method …
DZYNE Technologies（DZYNE技术公司）；Kitware, Inc.（Kitware公司）；University of Kentucky（肯塔基大学）
This work addresses the task of overhead image segmentation when auxiliary ground-level images are available. Recent work has shown that performing joint inference over these two modalities, often called near/remote sensing, can yield significant accuracy improvements. Extending this line of work, we introduce the concept of geospatial attention, a geometry-aware attention mechanism that explicitly considers the geospatial relationship between the pixels in a ground-level image and a geographic location. We propose an approach for computing geospatial attention that incorporates geometric feat…
Understanding why machine learning algorithms may fail is usually the task of the human expert that uses domain knowledge and contextual information to discover systematic shortcomings in either the data or the algorithm. In this paper, we propose a semantic referee, which is able to extract qualitative features of the errors emerging from deep machine learning frameworks and suggest corrections. The semantic referee relies on ontological reasoning about spatial knowledge in order to characterize errors in terms of their spatial relations with the environment. Using semantics, the reasoner int…
Harvard College（哈佛学院）；Defense Innovation Unit Experimental (DIUx)（国防创新实验单元）；DigitalGlobe（数字地球公司）；National Geospatial-Intelligence Agency (NGA)（国家地理空间情报局）
We introduce a new large-scale dataset for the advancement of object detection techniques and overhead object detection research. This satellite imagery dataset enables research progress pertaining to four key computer vision frontiers. We utilize a novel process for geospatial category detection and bounding box annotation with three stages of quality control. Our data is collected from WorldView-3 satellites at 0.3m ground sample distance, providing higher resolution imagery than most public satellite imagery datasets. We compare xView to other object detection datasets in both natural and o…
CrossEarth-SAR: A SAR-Centric and Billion-Scale Geospatial Foundation Model for Domain Generalizable Semantic Segmentation
Fudan University（复旦大学）；Shanghai Innovation Institute（上海创新研究院）；Shanghai Jiao Tong University（上海交通大学）；Beijing Institute of Technology（北京理工大学）；Wuhan University（武汉大学）；Zhongguancun Academy（中关村学院）；The University of Tokyo（东京大学）；Sun Yat-sen University（中山大学）；KU Leuven（比利时鲁汶大学）；KTH（皇家理工学院）
Beijing Institute of Technology（北京理工大学）；German Aerospace Center（德国航空航天中心）；Technical University of Munich（慕尼黑工业大学）；Univ. Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；CNRS（法国国家科学研究中心）；Grenoble INP（格勒诺布尔国立理工学院）；University of Iceland（冰岛大学）；Nanjing University of Science and Technology（南京理工大学）
Geospatial object detection of remote sensing imagery has been attracting an increasing interest in recent years, due to the rapid development in spaceborne imaging. Most of previously proposed object detectors are very sensitive to object deformations, such as scaling and rotation. To this end, we propose a novel and efficient framework for geospatial object detection in this letter, called Fourier-based rotation-invariant feature boosting (FRIFB). A Fourier-based rotation-invariant feature is first generated in polar coordinate. Then, the extracted features can be further structurally refine…
The performance of neural network models is often limited by the availability of big data sets. To treat this problem, we survey and develop novel synthetic data generation and augmentation techniques for enhancing low/zero-sample learning in satellite imagery. In addition to extending synthetic data generation approaches, we propose a hierarchical detection approach to improve the utility of synthetic training samples. We consider existing techniques for producing synthetic imagery--3D models and neural style transfer--as well as introducing our own adversarially trained reskinning network, t…
This paper proposes a novel approach to object detection on drone imagery, namely Multi-Proxy Detection Network with Unified Foreground Packing (UFPMP-Det). To deal with the numerous instances of very small scales, different from the common solution that divides the high-resolution input image into quite a number of chips with low foreground ratios to perform detection on them each, the Unified Foreground Packing (UFP) module is designed, where the sub-regions given by a coarse detector are initially merged through clustering to suppress background and the resulting ones are subsequently packe…
With the acceleration of the urban expansion, urban change detection (UCD), as a significant and effective approach, can provide the change information with respect to geospatial objects for dynamical urban analysis. However, existing datasets suffer from three bottlenecks: (1) lack of high spatial resolution images; (2) lack of semantic annotation; (3) lack of long-range multi-temporal images. In this paper, we propose a large scale benchmark dataset, termed Hi-UCD. This dataset uses aerial images with a spatial resolution of 0.1 m provided by the Estonia Land Board, including three-time phas…
FLAIR: a Country-Scale Land Cover Semantic Segmentation Dataset From Multi-Source Optical Imagery
French National Institute of Geographical and Forest Information (IGN)（法国国家地理与森林信息研究所（IGN））；Univ Gustave Eiffel（古斯塔夫·埃菲尔大学）；National School of Geographic Sciences (ENSG)（国家地理科学学院（ENSG））；Ecole des Ponts（巴黎桥路学院）；CNRS（法国国家科学研究中心）
Self-supervised representation learning techniques utilize large datasets without semantic annotations to learn meaningful, universal features that can be conveniently transferred to solve a wide variety of downstream supervised tasks. In this paper, we propose a self-supervised method for learning representations of geographic locations from unlabeled GPS trajectories to solve downstream geospatial computer vision tasks. Tiles resulting from a raster representation of the earth's surface are modeled as nodes on a graph or pixels of an image. GPS trajectories are modeled as allowed Markovian p…
Cloud and cloud shadow segmentation are fundamental processes in optical remote sensing image analysis. Current methods for cloud/shadow identification in geospatial imagery are not as accurate as they should, especially in the presence of snow and haze. This paper presents a deep learning-based framework for the detection of cloud/shadow in Landsat 8 images. Our method benefits from a convolutional neural network, Cloud-Net+ (a modification of our previously proposed Cloud-Net \cite{myigarss}) that is trained with a novel loss function (Filtered Jaccard Loss). The proposed loss function is mo…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Microsoft（微软）；University of Texas at San Antonio（德克萨斯大学圣安东尼奥分校）
Remotely sensed geospatial data are critical for applications including precision agriculture, urban planning, disaster monitoring and response, and climate change research, among others. Deep learning methods are particularly promising for modeling many remote sensing tasks given the success of deep neural networks in similar computer vision tasks and the sheer volume of remotely sensed imagery available. However, the variance in data collection methods and handling of geospatial metadata make the application of deep learning methodology to remotely sensed data nontrivial. For example, satell…
In response to the growing importance of geospatial data, its analysis including semantic segmentation becomes an increasingly popular task in computer vision today. Convolutional neural networks are powerful visual models that yield hierarchies of features and practitioners widely use them to process remote sensing data. When performing remote sensing image segmentation, multiple instances of one class with precisely defined boundaries are often the case, and it is crucial to extract those boundaries accurately. The accuracy of segments boundaries delineation influences the quality of the who…
SpectralEarth-FM: Bringing Hyperspectral Imagery into Multimodal Earth Observation Pretraining
Chair of Data Science in Earth Observation（地球观测数据科学主任）；Technical University of Munich（慕尼黑技术大学）；Remote Sensing Technology Institute（遥感技术研究所）；German Aerospace Center (DLR)（德国航空航天中心）；Department of Aerospace Engineering（航空航天工程系）；University of the Bundeswehr Munich（联邦国防军慕尼黑大学）；LEAP；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Univ. Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；Inria（法国国家信息与自动化技术研究院）；CNRS（法国国家科学研究中心）；Grenoble INP（格勒诺布尔INP）；LJK
Faculty of Mathematics and Computer Science（数学与计算机科学学院）
In recent years, the geospatial industry has been developing at a steady pace. This growth implies the addition of satellite constellations that produce a copious supply of satellite imagery and other Remote Sensing data on a daily basis. Sometimes, this information, even if in some cases we are referring to publicly available data, it sits unaccounted for due to the sheer size of it. Processing such large amounts of data with the help of human labour or by using traditional automation methods is not always a viable solution from the standpoint of both time and other resources. Within the pres…
Technical University of Denmark（丹麦技术大学）；Otovo AS（Otovo公司）
The body of research on classification of solar panel arrays from aerial imagery is increasing, yet there are still not many public benchmark datasets. This paper introduces two novel benchmark datasets for classifying and localizing solar panel arrays in Denmark: A human annotated dataset for classification and segmentation, as well as a classification dataset acquired using self-reported data from the Danish national building registry. We explore the performance of prior works on the new benchmark dataset, and present results after fine-tuning models using a similar approach as recent works.…
Cross-Modal Urban Sensing: Evaluating Sound-Vision Alignment Across Street-Level and Aerial Imagery
Department of Geography, University of South Carolina（南卡罗来纳大学地理系）；Department of Environmental Sciences, Emory University（埃默里大学环境科学系）；University of Canterbury（坎特伯雷大学）；School of Resource and Environmental Sciences, Wuhan University（武汉大学资源与环境科学学院）
Accurate and fine-grained information about the extent of damage to buildings is essential for directing Humanitarian Aid and Disaster Response (HADR) operations in the immediate aftermath of any natural calamity. In recent years, satellite and UAV (drone) imagery has been used for this purpose, sometimes aided by computer vision algorithms. Existing Computer Vision approaches for building damage assessment typically rely on a two stage approach, consisting of building detection using an object detection model, followed by damage assessment through classification of the detected building tiles…
Machine learning in remote sensing has matured alongside a proliferation in availability and resolution of geospatial imagery, but its utility is bottlenecked by the need for labeled data. What's more, many labeled geospatial datasets are specific to certain regions, instruments, or extreme weather events. We investigate the application of modern domain-adaptation to multiple proposed geospatial benchmarks, uncovering unique challenges and proposing solutions to them.
GeoRefer-Bench: A Benchmark from Referring Pixels to Verifiable Geospatial Reasoning
Central South University（中南大学）；Jiangxi Normal University（江西师范大学）；Hohai University（河海大学）；Renmin University of China（中国人民大学）；The Chinese University of Hong Kong（香港中文大学）
Decoding the Disaster: Multi-Task Geospatial Reasoning with Vision-Language Models and Crowdsourced Imagery for Disaster Mapping
Shandong University of Science and Technology（山东科技大学）；Technical University of Munich（慕尼黑工业大学）；Wuhan University（武汉大学）；Tsinghua University（清华大学）；Helmholtz-Zentrum Dresden-Rossendorf（亥姆霍兹德累斯顿罗森多夫研究中心）；University of Iceland（冰岛大学）
Self-supervised representation learning techniques utilize large datasets without semantic annotations to learn meaningful, universal features that can be conveniently transferred to solve a wide variety of downstream supervised tasks. In this work, we propose a self-supervised method for learning representations of geographic locations from unlabeled GPS trajectories to solve downstream geospatial computer vision tasks. Tiles resulting from a raster representation of the earth's surface are modeled as nodes on a graph or pixels of an image. GPS trajectories are modeled as allowed Markovian pa…
The shear volumes of data generated from earth observation and remote sensing technologies continue to make major impact; leaping key geospatial applications into the dual data and compute intensive era. As a consequence, this rapid advancement poses new computational and data processing challenges. We implement a novel remote sensing data flow (RESFlow) for advanced machine learning and computing with massive amounts of remotely sensed imagery. The core contribution is partitioning massive amount of data based on the spectral and semantic characteristics for distributed imagery analysis. RESF…
As a specific semantic segmentation task, aerial imagery segmentation has been widely employed in high spatial resolution (HSR) remote sensing images understanding. Besides common issues (e.g. large scale variation) faced by general semantic segmentation tasks, aerial imagery segmentation has some unique challenges, the most critical one among which lies in foreground-background imbalance. There have been some recent efforts that attempt to address this issue by proposing sophisticated neural network architectures, since they can be used to extract informative multi-scale feature representatio…
HyBiomass: Global Hyperspectral Imagery Benchmark Dataset for Evaluating Geospatial Foundation Models in Forest Aboveground Biomass Estimation
Remote Sensing Technology Institute, German Aerospace Center (DLR)（远程传感技术研究所，德国航天中心）；Global Land Monitoring Group, Helmholtz German Research Centre for Geosciences (GFZ)（全球土地监测组，海德堡德国地理科学研究中心）；Department of Aerospace Engineering, University of the Bundeswehr Munich（航空航天工程系，联邦国防军 Munich 大学）；Chair of Data Science in Earth Observation (SiPEO), Technical University of Munich (TUM)（地球观测数据科学教授职位（SiPEO），慕尼黑技术大学）
Advancing Earth Observation Through Machine Learning: A TorchGeo Tutorial
Microsoft AI for Good Research Lab（微软AI for Good研究实验室）；Chair of Data Science in Earth Observation, Technical University of Munich（地球观测数据科学主任，技术大学慕尼黑）；Chair of Earth Observation, University of the Bundeswehr Munich（地球观测主任，联邦国防军大学慕尼黑）；Department of Robotics, Perception and Learning, KTH Royal Institute of Technology（机器人、感知与学习系，皇家理工学院）；Wherobots（Wherobots公司）；National Water and Sanitation Agency, Brazil, Federal District, Brasilia（巴西国家水务与卫生局，联邦区，巴西利亚）
GeoViSTA: Geospatial Vision-Tabular Transformer for Multimodal Environment Representation
Department of Electrical and Computer Engineering, Rice University（理海大学电气与计算机工程系）；Center for Cardiovascular Computational and Precision Health, Department of Cardiology, DeBakey Heart and Vascular Center, Houston Methodist（休斯顿方法主义医疗中心心血管计算与精准健康中心、心内科部门、德贝基心脏和血管中心）
Accurately maintaining digital street maps is labor-intensive. To address this challenge, much work has studied automatically processing geospatial data sources such as GPS trajectories and satellite images to reduce the cost of maintaining digital maps. An end-to-end map update system would first process geospatial data sources to extract insights, and second leverage those insights to update and improve the map. However, prior work largely focuses on the first step of this pipeline: these map extraction methods infer road networks from scratch given geospatial data sources (in effect creatin…
University of Southern California（南加州大学）；Microsoft（微软）
Recent work has shown that deep learning models can be used to classify land-use data from geospatial satellite imagery. We show that when these deep learning models are trained on data from specific continents/seasons, there is a high degree of variability in model performance on out-of-sample continents/seasons. This suggests that just because a model accurately predicts land-use classes in one continent or season does not mean that the model will accurately predict land-use classes in a different continent or season. We then use clustering techniques on satellite imagery from different cont…
