Self-supervision based deep learning classification approaches have received considerable attention in academic literature. However, the performance of such methods on remote sensing imagery domains remains under-explored. In this work, we explore contrastive representation learning methods on the task of imagery-based city classification, an important problem in urban computing. We use satellite and map imagery across 2 domains, 3 million locations and more than 1500 cities. We show that self-supervised methods can build a generalizable representation from as few as 200 cities, with representations achieving over 95\% accuracy in unseen cities with minimal additional training. We also find that the performance discrepancy of such methods, when compared to supervised methods, induced by the domain discrepancy between natural imagery and abstract imagery is significant for remote sensing imagery. We compare all analysis against existing supervised models from academic literature and open-source our models for broader usage and further criticism.
In recent years, supervised learning has been widely used in various tasks of optical remote sensing image understanding, including remote sensing image classification, pixel-wise segmentation, change detection, and object detection. The methods based on supervised learning need a large amount of high-quality training data and their performance highly depends on the quality of the labels. However, in practical remote sensing applications, it is often expensive and time-consuming to obtain large-scale data sets with high-quality labels, which leads to a lack of sufficient supervised information…
Bridging Supervision Gaps: A Unified Framework for Remote Sensing Change Detection
State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan, China（测绘遥感信息工程国家重点实验室，武汉大学，中国）；The University of Tokyo, Tokyo, Japan（东京大学，日本）
GeoSolver: Scaling Test-Time Reasoning in Remote Sensing with Fine-Grained Process Supervision
College of Computer Science and Technology（计算机科学与技术学院）；Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education Jilin University（教育部符号计算与知识工程重点实验室）
For change detection in remote sensing, constructing a training dataset for deep learning models is difficult due to the requirements of bi-temporal supervision. To overcome this issue, single-temporal supervision which treats change labels as the difference of two semantic masks has been proposed. This novel method trains a change detector using two spatially unrelated images with corresponding semantic labels such as building. However, training on unpaired datasets could confuse the change detector in the case of pixels that are labeled unchanged but are visually significantly different. In …
The proliferation of remote sensing satellites has resulted in a massive amount of remote sensing images. However, due to human and material resource constraints, the vast majority of remote sensing images remain unlabeled. As a result, it cannot be applied to currently available deep learning methods. To fully utilize the remaining unlabeled images, we propose a Geographical Knowledge-driven Representation learning method for remote sensing images (GeoKR), improving network performance and reduce the demand for annotated data. The global land cover products and geographical location associate…
Expediting Building Footprint Extraction from High-resolution Remote Sensing Images via progressive lenient supervision
State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University（武汉大学测绘遥感信息工程国家重点实验室）；School of Remote Sensing and Information Engineering, Wuhan University（武汉大学遥感信息工程学院）；National Engineering Research Center for Multimedia Software, Wuhan University（武汉大学国家多媒体软件工程技术研究中心）；Institute of Artificial Intelligence, Wuhan University（武汉大学人工智能研究院）；School of Computer Science, Wuhan University（武汉大学计算机学院）
iSAGE: A Human-in-the-Loop Framework for Remote Sensing Semantic Segmentation via Sparse Point Supervision
Department of Electrical Engineering, University of Bras\'ilia, Bras\'ilia, DF, Brazil；Department of Geography, University of Bras\'ilia, Bras\'ilia, DF, Brazil
GeoSearcher: Anchor-Guided Progressive Reasoning for Remote Sensing Visual Grounding with Process Supervision
Key Laboratory of Target Cognition and Application Technology (TCAT), Chinese Academy of Sciences（中国科学院目标认知与应用技术重点实验室）；School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences（中国科学院大学电子电气与通信工程学院）
Beihang University（北京航空航天大学）；Jeonbuk National University（全北国立大学）
Supervised deep learning models depend on massive labeled data. Unfortunately, it is time-consuming and labor-intensive to collect and annotate bitemporal samples containing desired changes. Transfer learning from pre-trained models is effective to alleviate label insufficiency in remote sensing (RS) change detection (CD). We explore the use of semantic information during pre-training. Different from traditional supervised pre-training that learns the mapping from image to label, we incorporate semantic supervision into the self-supervised learning (SSL) framework. Typically, multiple objects …
In this paper, we focus on the challenging multicategory instance segmentation problem in remote sensing images (RSIs), which aims at predicting the categories of all instances and localizing them with pixel-level masks. Although many landmark frameworks have demonstrated promising performance in instance segmentation, the complexity in the background and scale variability instances still remain challenging for instance segmentation of RSIs. To address the above problems, we propose an end-to-end multi-category instance segmentation model, namely Semantic Attention and Scale Complementary Netw…
Huazhong University of Science and Technology（华中科技大学）；Qingdao University（青岛大学）
Detecting oriented objects along with estimating their rotation information is one crucial step for analyzing remote sensing images. Despite that many methods proposed recently have achieved remarkable performance, most of them directly learn to predict object directions under the supervision of only one (e.g. the rotation angle) or a few (e.g. several coordinates) groundtruth values individually. Oriented object detection would be more accurate and robust if extra constraints, with respect to proposal and rotation information regression, are adopted for joint supervision during training. To t…
MonoIR-RS: Infrared Remote Sensing Vision-Language Learning with CLIP and VLM Adaptation
China University of Petroleum-Beijing at Karamay（中国石油大学（北京）克拉玛依校区）；Guizhou University（贵州大学）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；Shanghai Jiao Tong University（上海交通大学）
The binary segmentation of roads in very high resolution (VHR) remote sensing images (RSIs) has always been a challenging task due to factors such as occlusions (caused by shadows, trees, buildings, etc.) and the intra-class variances of road surfaces. The wide use of convolutional neural networks (CNNs) has greatly improved the segmentation accuracy and made the task end-to-end trainable. However, there are still margins to improve in terms of the completeness and connectivity of the results. In this paper, we consider the specific context of road extraction and present a direction-aware resi…
Center-guided Classifier for Semantic Segmentation of Remote Sensing Images
School of Software Technology, Zhejiang University（浙江大学软件学院）；Innovation Center of Yangtze River Delta, Zhejiang University（浙江大学长三角创新中心）；School of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；City University of Hong Kong（香港城市大学）
Evaluating Semantic and Spatial Guidance for Foundation Model Segmentation of Small-Scale PV in Remote Sensing Imagery
Ben-Gurion University of the Negev（内盖夫本-古里安大学）；The Hebrew University of Jerusalem（耶路撒冷希伯来大学）；Ben Gurion Institute for the Study of Israel & Zionism（本-古里安以色列与犹太复国主义研究所）；Ben-Gurion Israel Research Institute（本-古里安以色列研究所）
Harbin Institute of Technology（哈尔滨工业大学）；Beijing Orient Institute of Measurement and Test（北京东方计量测试研究所）
Recently, CNN object detectors have achieved high accuracy on remote sensing images but require huge labor and time costs on annotation. In this paper, we propose a new uncertainty-based active learning which can select images with more information for annotation and detector can still reach high performance with a fraction of the training images. Our method not only analyzes objects' classification uncertainty to find least confident objects but also considers their regression uncertainty to declare outliers. Besides, we bring out two extra weights to overcome two difficulties in remote sensi…
FusionRS: A Large-Scale RGB-Infrared-Style Remote Sensing Dataset for Cross-Modal Vision-Language Learning
China University of Petroleum-Beijing at Karamay（中国石油大学（北京）克拉玛依校区）；University of Electronic Science and Technology of China（电子科技大学）；Tianjin University（天津大学）
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Idiap Research Institute（伊迪亚普研究所）；NVIDIA（英伟达）；University of Toronto（多伦多大学）；Vector Institute（矢量研究所）
Impressive progress in 3D shape extraction led to representations that can capture object geometries with high fidelity. In parallel, primitive-based methods seek to represent objects as semantically consistent part arrangements. However, due to the simplicity of existing primitive representations, these methods fail to accurately reconstruct 3D shapes using a small number of primitives/parts. We address the trade-off between reconstruction quality and number of parts with Neural Parts, a novel 3D primitive representation that defines primitives using an Invertible Neural Network (INN) which i…
Self-supervised learning aims to learn image feature representations without the usage of manually annotated labels. It is often used as a precursor step to obtain useful initial network weights which contribute to faster convergence and superior performance of downstream tasks. While self-supervision allows one to reduce the domain gap between supervised and unsupervised learning without the usage of labels, the self-supervised objective still requires a strong inductive bias to downstream tasks for effective transfer learning. In this work, we present our material and texture based self-supe…
Lahore University of Management Sciences（拉合尔管理科学大学）
The advent of deep perceptual networks brought about a paradigm shift in machine vision and image perception. Image apprehension lately carried out by hand-crafted features in the latent space have been replaced by deep features acquired from supervised networks for improved understanding. However, such deep networks require strict supervision with a substantial amount of the labeled data for authentic training process. These methods perform poorly in domains lacking labeled data especially in case of remote sensing image retrieval. Resolving this, we propose an unsupervised encoder-decoder fe…
Any2Any: Unified Arbitrary Modality Translation for Remote Sensing
National Engineering Research Center for Multimedia Software, Institute of Artificial Intelligence, School of Computer Science, Wuhan University（多媒体软件国家工程研究中心、人工智能研究院、计算机科学学院、武汉大学）；Hubei Key Laboratory of Multimedia（湖北省多媒体重点实验室）；Zhongguancun Academy, Beijing, China. 100094（中关村学院，北京，中国。100094）；School of Electronic Information, Wuhan University, Wuhan, China（电子信息学院，武汉大学，武汉，中国）；School of Automation, Beijing Institute of Technology（自动化学院，北京理工大学）；State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan, China（测绘、制图与遥感信息工程国家重点实验室，武汉大学，武汉，中国）
Object counting, which aims to count the accurate number of object instances in images, has been attracting more and more attention. However, challenges such as large scale variation, complex background interference, and non-uniform density distribution greatly limit the counting accuracy, particularly striking in remote sensing imagery. To mitigate the above issues, this paper proposes a novel framework for dense object counting in remote sensing images, which incorporates a pyramidal scale module (PSM) and a global context module (GCM), dubbed PSGCNet, where PSM is used to adaptively capture…
University of California Berkeley（加州大学伯克利分校）；Stanford University（斯坦福大学）；Massachusetts Institute of Technology（麻省理工学院）
This paper tests the hypothesis that modeling a scene in terms of entities and their local interactions, as opposed to modeling the scene globally, provides a significant benefit in generalizing to physical tasks in a combinatorial space the learner has not encountered before. We present object-centric perception, prediction, and planning (OP3), which to the best of our knowledge is the first fully probabilistic entity-centric dynamic latent variable framework for model-based reinforcement learning that acquires entity representations from raw visual observations without supervision and uses t…
GeoAlignCLIP: Enhancing Fine-Grained Vision-Language Alignment in Remote Sensing via Multi-Granular Consistency Learning
College of Computer Science and Technology, Jilin University, Changchun 130012, China（吉林大学计算机科学与技术学院，长春130012，中国）；Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education Jilin University（教育部符号计算与知识工程重点实验室，吉林大学）
Prototype-Based Pseudo-Label Denoising for Source-Free Domain Adaptation in Remote Sensing Semantic Segmentation
College of Computer Science, Sichuan University（四川大学计算机科学学院）；College of Computer Science and Cyber Security, Chengdu University of Technology（成都理工大学计算机科学与网络安全学院）
NEC Laboratories America（NEC美国实验室）；UC San Diego（加州大学圣地亚哥分校）
We propose an end-to-end network that takes a single perspective RGB image of a complex road scene as input, to produce occlusion-reasoned layouts in perspective space as well as a parametric bird's-eye-view (BEV) space. In contrast to prior works that require dense supervision such as semantic labels in perspective view, our method only requires human annotations for parametric attributes that are cheaper and less ambiguous to obtain. To solve this challenging task, our design is comprised of modules that incorporate inductive biases to learn occlusion-reasoning, geometric transformation and …
This paper introduces the use of single layer and deep convolutional networks for remote sensing data analysis. Direct application to multi- and hyper-spectral imagery of supervised (shallow or deep) convolutional networks is very challenging given the high input data dimensionality and the relatively small amount of available labeled data. Therefore, we propose the use of greedy layer-wise unsupervised pre-training coupled with a highly efficient algorithm for unsupervised learning of sparse features. The algorithm is rooted on sparse representations and enforces both population and lifetime …
GeoMeld: Toward Semantically Grounded Foundation Models for Remote Sensing
Indian Institute of Technology Bombay（印度理工学院孟买分校）；Space Applications Centre, ISRO（印度空间研究组织空间应用中心）；University of Trento（特伦托大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
New York University（纽约大学）；NYU Abu Dhabi（纽约大学阿布扎比分校）
Recent studies have shown the benefits of using additional elevation data (e.g., DSM) for enhancing the performance of the semantic segmentation of aerial images. However, previous methods mostly adopt 3D elevation information as additional inputs. While in many real-world applications, one does not have the corresponding DSM information at hand and the spatial resolution of acquired DSM images usually do not match the aerial images. To alleviate this data constraint and also take advantage of 3D elevation information, in this paper, we introduce a geometry-aware segmentation model that achiev…
