RSI-Net: Two-Stream Deep Neural Network for Remote Sensing Imagesbased Semantic Segmentation
Jiangsu Ocean University（江苏海洋大学）；Dalhousie University（达尔豪斯大学）；Nanjing University of Information Science and Technology（南京信息工程大学）；Shenzhen University（深圳大学）；Peng Cheng Laboratory（鹏城实验室）；ShanghaiTech University（上海科技大学）
Shuang He, Xia Lu, Jason Gu, Haitong Tang, Qin Yu, Kaiyue Liu, Haozhou Ding, Chunqi Chang, Nizhuan Wang
For semantic segmentation of remote sensing images (RSI), trade-off between representation power and location accuracy is quite important. How to get the trade-off effectively is an open question,where current approaches of utilizing very deep models result in complex models with large memory consumption. In contrast to previous work that utilizes dilated convolutions or deep models, we propose a novel two-stream deep neural network for semantic segmentation of RSI (RSI-Net) to obtain improved performance through modeling and propagating spatial contextual structure effectively and a decoding scheme with image-level and graph-level combination. The first component explicitly models correlations between adjacent land covers and conduct flexible convolution on arbitrarily irregular image regions by using graph convolutional network, while densely connected atrous convolution network (DenseAtrousCNet) with multi-scale atrous convolution can expand the receptive fields and obtain image global information. Extensive experiments are implemented on the Vaihingen, Potsdam and Gaofen RSI datasets, where the comparison results demonstrate the superior performance of RSI-Net in terms of overall accuracy (91.83%, 93.31% and 93.67% on three datasets, respectively), F1 score (90.3%, 91.49% and 89.35% on three datasets, respectively) and kappa coefficient (89.46%, 90.46% and 90.37% on three datasets, respectively) when compared with six state-of-the-art RSI semantic segmentation methods.
Austrian Institute of Technology（奥地利技术研究所）；University of Science and Technology, University of Danang（岘港大学科技大学）；University of Essex（埃塞克斯大学）
The task of remote sensing image scene classification (RSISC), which aims at classifying remote sensing images into groups of semantic categories based on their contents, has taken the important role in a wide range of applications such as urban planning, natural hazards detection, environment monitoring,vegetation mapping, or geospatial object detection. During the past years, research community focusing on RSISC task has shown significant effort to publish diverse datasets as well as propose different approaches to deal with the RSISC challenges. Recently, almost proposed RSISC systems base …
Central South University（中南大学）；Wuhan University（武汉大学）；Southwest Jiaotong University（西南交通大学）
Do we on the right way for remote sensing image understanding (RSIU) by training models via supervised data-dependent and task-dependent way, instead of human vision in a label-free and task-independent way? We argue that a more desirable RSIU model should be trained with intrinsic structure from data rather that extrinsic human labels to realize generalizability across a wide range of RSIU tasks. According to this hypothesis, we proposed \textbf{T}he \textbf{O}riginal \textbf{V}ision model (TOV) in remote sensing filed. Trained by massive unlabeled optical data along a human-like self-supervi…
Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；University of Leicester（莱斯特大学）
Due to the more dramatic multi-scale variations and more complicated foregrounds and backgrounds in optical remote sensing images (RSIs), the salient object detection (SOD) for optical RSIs becomes a huge challenge. However, different from natural scene images (NSIs), the discussion on the optical RSI SOD task still remains scarce. In this paper, we propose a multi-scale context network, namely MSCNet, for SOD in optical RSIs. Specifically, a multi-scale context extraction module is adopted to address the scale variation of salient objects by effectively learning multi-scale contextual informa…
Salient object detection (SOD) in optical remote sensing images (RSIs), or RSI-SOD, is an emerging topic in understanding optical RSIs. However, due to the difference between optical RSIs and natural scene images (NSIs), directly applying NSI-SOD methods to optical RSIs fails to achieve satisfactory results. In this paper, we propose a novel Adjacent Context Coordination Network (ACCoNet) to explore the coordination of adjacent features in an encoder-decoder architecture for RSI-SOD. Specifically, ACCoNet consists of three parts: an encoder, Adjacent Context Coordination Modules (ACCoMs), and …
University of Electronic Science and Technology of China（电子科技大学）；Truly Opto-Electronics Co., Ltd.（信利光电股份有限公司）
The Siamese network is becoming the mainstream in change detection of remote sensing images (RSI). However, in recent years, the development of more complicated structure, module and training processe has resulted in the cumbersome model, which hampers their application in large-scale RSI processing. To this end, this paper proposes an extremely lightweight Siamese network (LSNet) for RSI change detection, which replaces standard convolution with depthwise separable atrous convolution, and removes redundant dense connections, retaining only valid feature flows while performing Siamese feature …
Shanghai Institute for Advanced Communication and Data Science, Shanghai University（上海大学上海先进通信与数据科学研究院）；School of Communication and Information Engineering, Shanghai University（上海大学通信与信息工程学院）；School of Computer Science and Engineering, Nanyang Technological University（南洋理工大学计算机科学与工程学院）；Department of Computer Science, Stony Brook University（石溪大学计算机科学系）
In the computer vision community, great progresses have been achieved in salient object detection from natural scene images (NSI-SOD); by contrast, salient object detection in optical remote sensing images (RSI-SOD) remains to be a challenging emerging topic. The unique characteristics of optical RSIs, such as scales, illuminations and imaging orientations, bring significant differences between NSI-SOD and RSI-SOD. In this paper, we propose a novel Multi-Content Complementation Network (MCCNet) to explore the complementarity of multiple content for RSI-SOD. Specifically, MCCNet is based on the…
The existing SSCL of RSI is built based on constructing positive and negative sample pairs. However, due to the richness of RSI ground objects and the complexity of the RSI contextual semantics, the same RSI patches have the coexistence and imbalance of positive and negative samples, which causing the SSCL pushing negative samples far away while pushing positive samples far away, and vice versa. We call this the sample confounding issue (SCI). To solve this problem, we propose a False negAtive sampLes aware contraStive lEarning model (FALSE) for the semantic segmentation of high-resolution RSI…
From Pixels to Images: A Structural Survey of Deep Learning Paradigms in Remote Sensing Image Semantic Segmentation
College of Science and Engineering and Centre for AI and Data Science Innovation, James Cook University（科学与工程学院和人工智能与数据科学创新中心，詹姆斯库克大学）；Department of Forest and Wildlife Ecology, University of Wisconsin-Madison（森林与野生动物生态学系，威斯康星大学麦迪逊分校）；School of Computing, Engineering and Mathematical Sciences, La Trobe University（计算、工程与数学科学学院，拉特罗布大学）
Beijing Jiaotong University（北京交通大学）；City University of Hong Kong（香港城市大学）；Hunan University（湖南大学）；Peng Cheng Laboratory（鹏城实验室）；Sun Yat-sen University（中山大学）；City University of Hong Kong Shenzhen Research Institute（香港城市大学深圳研究院）
Salient object detection (SOD) for optical remote sensing images (RSIs) aims at locating and extracting visually distinctive objects/regions from the optical RSIs. Despite some saliency models were proposed to solve the intrinsic problem of optical RSIs (such as complex background and scale-variant objects), the accuracy and completeness are still unsatisfactory. To this end, we propose a relational reasoning network with parallel multi-scale attention for SOD in optical RSIs in this paper. The relational reasoning module that integrates the spatial and the channel dimensions is designed to in…
City University of Hong Kong（香港城市大学）；Beijing Jiaotong University（北京交通大学）；City University of Hong Kong Shenzhen Research Institute（香港城市大学深圳研究院）；Tianjin University（天津大学）
Arising from the various object types and scales, diverse imaging orientations, and cluttered backgrounds in optical remote sensing image (RSI), it is difficult to directly extend the success of salient object detection for nature scene image to the optical RSI. In this paper, we propose an end-to-end deep network called LV-Net based on the shape of network architecture, which detects salient objects from optical RSIs in a purely data-driven fashion. The proposed LV-Net consists of two key modules, i.e., a two-stream pyramid module (L-shaped module) and an encoder-decoder module with nested co…
Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；University of Leicester（莱斯特大学）
Due to the extreme complexity of scale and shape as well as the uncertainty of the predicted location, salient object detection in optical remote sensing images (RSI-SOD) is a very difficult task. The existing SOD methods can satisfy the detection performance for natural scene images, but they are not well adapted to RSI-SOD due to the above-mentioned image characteristics in remote sensing images. In this paper, we propose a novel Attention Guided Network (AGNet) for SOD in optical RSIs, including position enhancement stage and detail refinement stage. Specifically, the position enhancement s…
Insight Any Instance: Promptable Instance Segmentation for Remote Sensing Images
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；University of Chinese Academy of Sciences（中国科学院大学）；School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences（中国科学院大学电子电气与通信工程学院）
Remote Sensing Change Detection (RS-CD) aims to detect relevant changes from Multi-Temporal Remote Sensing Images (MT-RSIs), which aids in various RS applications such as land cover, land use, human development analysis, and disaster response. The performance of existing RS-CD methods is attributed to training on large annotated datasets. Furthermore, most of these models are less transferable in the sense that the trained model often performs very poorly when there is a domain gap between training and test datasets. This paper proposes an unsupervised CD method based on deep metric learning t…
Adapting Segment Anything Model for Change Detection in HR Remote Sensing Images
Information Engineering University（信息工程大学）；Nanjing University of Information Science and Technology（南京信息工程大学）；ETH Zurich（苏黎世联邦理工学院）；University of Trento（特伦托大学）
City University of Hong Kong（香港城市大学）；Beijing Jiaotong University（北京交通大学）
Despite the remarkable advances in visual saliency analysis for natural scene images (NSIs), salient object detection (SOD) for optical remote sensing images (RSIs) still remains an open and challenging problem. In this paper, we propose an end-to-end Dense Attention Fluid Network (DAFNet) for SOD in optical RSIs. A Global Context-aware Attention (GCA) module is proposed to adaptively capture long-range semantic context relationships, and is further embedded in a Dense Attention Fluid (DAF) structure that enables shallow attention cues flow into deep layers to guide the generation of high-leve…
School of Electrical and Information Engineering, Tianjin University（天津大学电气与信息工程学院）；Institute of Information Science, Beijing Jiaotong University（北京交通大学信息科学研究所）；Beijing Key Laboratory of Advanced Information Science and Network Technology（北京市先进信息科学与网络技术重点实验室）；School of Software Engineering, Huazhong University of Science and Technology（华中科技大学软件工程学院）；Department of Computer Science, City University of Hong Kong（香港城市大学计算机科学系）；Department of Computer Science and Engineering, Southern University of Science and Technology（南方科技大学计算机科学与工程系）
The diverse spatial resolutions, various object types, scales and orientations, and cluttered backgrounds in optical remote sensing images (RSIs) challenge the current salient object detection (SOD) approaches. It is commonly unsatisfactory to directly employ the SOD approaches designed for nature scene images (NSIs) to RSIs. In this paper, we propose a novel Parallel Down-up Fusion network (PDF-Net) for SOD in optical RSIs, which takes full advantage of the in-path low- and high-level features and cross-path multi-resolution features to distinguish diversely scaled salient objects and suppres…
Although domain adaptation has been extensively studied in natural image-based segmentation task, the research on cross-domain segmentation for very high resolution (VHR) remote sensing images (RSIs) still remains underexplored. The VHR RSIs-based cross-domain segmentation mainly faces two critical challenges: 1) Large area land covers with many diverse object categories bring severe local patch-level data distribution deviations, thus yielding different adaptation difficulties for different local patches; 2) Different VHR sensor types or dynamically changing modes cause the VHR images to go t…
PLA Strategic Force Information Engineering University（中国人民解放军战略支援部队信息工程大学）；Space Engineering University（航天工程大学）；State Key Laboratory of Geo-Information Engineering（地理信息工程国家重点实验室）；Beijing University of Technology（北京工业大学）；University of Trento（特伦托大学）；Beijing Institute of Remote Sensing Information（北京遥感信息研究所）；China University of Geosciences (Beijing)（中国地质大学（北京））；ETH Zurich（苏黎世联邦理工学院）
Long-range contextual information is crucial for the semantic segmentation of High-Resolution (HR) Remote Sensing Images (RSIs). However, image cropping operations, commonly used for training neural networks, limit the perception of long-range contexts in large RSIs. To overcome this limitation, we propose a Wide-Context Network (WiCoNet) for the semantic segmentation of HR RSIs. Apart from extracting local features with a conventional CNN, the WiCoNet has an extra context branch to aggregate information from a larger image area. Moreover, we introduce a Context Transformer to embed contextual…
Shenzhen University（深圳大学）；The University of Nottingham（诺丁汉大学）；Shenzhen Institute of AI and Robotics for Society（深圳人工智能与机器人研究院）；Pengcheng Laboratory（鹏城实验室）
In supervised deep learning, learning good representations for remote--sensing images (RSI) relies on manual annotations. However, in the area of remote sensing, it is hard to obtain huge amounts of labeled data. Recently, self--supervised learning shows its outstanding capability to learn representations of images, especially the methods of instance discrimination. Comparing methods of instance discrimination, clustering--based methods not only view the transformations of the same image as ``positive" samples but also similar images. In this paper, we propose a new clustering-based method for…
PointSAM: Pointly-Supervised Segment Anything Model for Remote Sensing Images
School of Information Science and Technology, Southwest Jiaotong University（西南交通大学信息科学与技术学院）；I2R, A*STAR（新加坡科研局信息研究院）；South China University of Technology（华南理工大学）
In this paper, we focus on the challenging multicategory instance segmentation problem in remote sensing images (RSIs), which aims at predicting the categories of all instances and localizing them with pixel-level masks. Although many landmark frameworks have demonstrated promising performance in instance segmentation, the complexity in the background and scale variability instances still remain challenging for instance segmentation of RSIs. To address the above problems, we propose an end-to-end multi-category instance segmentation model, namely Semantic Attention and Scale Complementary Netw…
FedEU: Evidential Uncertainty-Driven Federated Fine-Tuning of Vision Foundation Models for Remote Sensing Image Segmentation
School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院）；School of Information Science and Engineering, Wuhan University of Science and Technology（武汉科技大学信息科学与工程学院）；School of Computer Science, Wuhan University（武汉大学计算机学院）
Center-guided Classifier for Semantic Segmentation of Remote Sensing Images
School of Software Technology, Zhejiang University（浙江大学软件学院）；Innovation Center of Yangtze River Delta, Zhejiang University（浙江大学长三角创新中心）；School of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；City University of Hong Kong（香港城市大学）
Austrian Institute of Technology（奥地利技术研究所）；HCM University of Technology（胡志明市理工大学）；University of Essex（埃塞克斯大学）；FPT Soft Company（FPT软件公司）；Singapore Institute of Technology（新加坡理工大学）
In this paper, we present a high-performance and light-weight deep learning model for Remote Sensing Image Classification (RSIC), the task of identifying the aerial scene of a remote sensing image. To this end, we first valuate various benchmark convolutional neural network (CNN) architectures: MobileNet V1/V2, ResNet 50/151V2, InceptionV3/InceptionResNetV2, EfficientNet B0/B7, DenseNet 121/201, ConNeXt Tiny/Large. Then, the best performing models are selected to train a compact model in a teacher-student arrangement. The knowledge distillation from the teacher aims to achieve high performance…
Align, Perturb and Decouple: Toward Better Leverage of Difference Information for RSI Change Detection
Fudan University（复旦大学）；Tencent（腾讯）；Zhengzhou Zhongke Institute of Integrated circuit and System Application（郑州中科集成电路与系统应用研究院）；Zhongshan PoolNet Technology Ltd（中山普尔网络科技有限公司）；Shanghai Jiao Tong University（上海交通大学）；Rongcheer（融畅）
College of Electronics and Information Engineering, Sichuan University（四川大学电子信息工程学院）；School of Mechanical Engineering and Automation, Fuzhou University（福州大学机械工程与自动化学院）；Yunnan Key Laboratory of Software Engineering, Yunnan University（云南大学云南省软件工程重点实验室）
PKINet-v2: Towards Powerful and Efficient Poly-Kernel Remote Sensing Object Detection
Nanjing University of Science and Technology（南京理工大学）；Zhejiang University（浙江大学）；Department of Electrical and Computer Engineering, Sungkyunkwan University（成均馆大学电子与计算机工程系）；State Key Lab. of Intelligent Manufacturing of Advanced Construction Machinery（先进施工机械智能制造国家重点实验室）
Change-Agent: Towards Interactive Comprehensive Remote Sensing Change Interpretation and Analysis
Beihang University（北京航空航天大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；School of Astronautics, Beihang University（北京航空航天大学宇航学院）；Shen Yuan Honors College of Beihang University（北京航空航天大学沈元荣誉学院）
With the wide application of remote sensing technology in various fields, the accuracy and security requirements for remote sensing images (RSIs) recognition are also increasing. In recent years, due to the rapid development of deep learning in the field of image recognition, RSI recognition models based on deep convolution neural networks (CNNs) outperform traditional hand-craft feature techniques. However, CNNs also pose security issues when they show their capability of accurate classification. By adding a very small variation of the adversarial perturbation to the input image, the CNN mode…
RSICCLLM: A Multimodal Large Language Model for Remote Sensing Image Change Captioning
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China（人工智能安全国家重点实验室，计算技术研究所，中国科学院，北京，中国）；Great Bay University, Dongguan, China（东莞Great Bay大学，中国）；Harbin Institute of Technology, Shenzhen, China（哈尔滨工业大学深圳学院，中国）；Dongguan Key Laboratory for Intelligence and Information Technology, Dongguan, China（东莞智能与信息科技重点实验室，中国）
Prototype-Based Pseudo-Label Denoising for Source-Free Domain Adaptation in Remote Sensing Semantic Segmentation
College of Computer Science, Sichuan University（四川大学计算机科学学院）；College of Computer Science and Cyber Security, Chengdu University of Technology（成都理工大学计算机科学与网络安全学院）
Anhui University（安徽大学）；Shanghai University（上海大学）；East China Normal University（华东师范大学）
Recently, with the application of deep learning in the remote sensing image (RSI) field, the classification accuracy of the RSI has been dramatically improved compared with traditional technology. However, even the state-of-the-art object recognition convolutional neural networks are fooled by the universal adversarial perturbation (UAP). The research on UAP is mostly limited to ordinary images, and RSIs have not been studied. To explore the basic characteristics of UAPs of RSIs, this paper proposes a novel method combining an encoder-decoder network with an attention mechanism to generate the…
A Survey of Sample-Efficient Deep Learning for Change Detection in Remote Sensing: Tasks, Strategies, and Challenges
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；Information Engineering University（信息工程大学）；School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences（中国科学院大学电子电气与通信工程学院）；School of Mathematics and Statistics, Southeast University（东南大学数学学院）；Graduate School of Frontier Sciences, The University of Tokyo（东京大学前沿科学研究生院）；RIKEN Center for Advanced Intelligence Project（理化学研究所先进智能项目中心）；Department of Information Engineering and Computer Science, University of Trento（特伦托大学信息工程与计算机科学系）
Wuhan University（武汉大学）；University of California, Merced（加州大学默塞德分校）
Remote sensing image retrieval(RSIR), which aims to efficiently retrieve data of interest from large collections of remote sensing data, is a fundamental task in remote sensing. Over the past several decades, there has been significant effort to extract powerful feature representations for this task since the retrieval performance depends on the representative strength of the features. Benchmark datasets are also critical for developing, evaluating, and comparing RSIR approaches. Current benchmark datasets are deficient in that 1) they were originally collected for land use/land cover classifi…
Ocean University of China（中国海洋大学）；Mississippi State University（密西西比州立大学）
Existing remote sensing change detection methods are heavily affected by seasonal variation. Since vegetation colors are different between winter and summer, such variations are inclined to be falsely detected as changes. In this letter, we proposed an image translation method to solve the problem. A style-based recalibration module is introduced to capture seasonal features effectively. Then, a new style discriminator is designed to improve the translation performance. The discriminator can not only produce a decision for the fake or real sample, but also return a style vector according to th…
The collection of a high number of pixel-based labeled training samples for tree species identification is time consuming and costly in operational forestry applications. To address this problem, in this paper we investigate the effectiveness of explanation methods for deep neural networks in performing weakly supervised semantic segmentation using only image-level labels. Specifically, we consider four methods:i) class activation maps (CAM); ii) gradient-based CAM; iii) pixel correlation module; and iv) self-enhancing maps (SEM). We compare these methods with each other using both quantitativ…
Recently, a massive number of deep learning based approaches have been successfully applied to various remote sensing image (RSI) recognition tasks. However, most existing advances of deep learning methods in the RSI field heavily rely on the features extracted by the manually designed backbone network, which severely hinders the potential of deep learning models due the complexity of RSI and the limitation of prior knowledge. In this paper, we research a new design paradigm for the backbone architecture in RSI recognition tasks, including scene classification, land-cover classification and ob…
University of Electronic Science and Technology of China（电子科技大学）；Henan University（河南大学）
Remote sensing image (RSI) inpainting plays an important role in real applications. Recently, fully-connected tensor network (FCTN) decomposition has been shown the remarkable ability to fully characterize the global correlation. Considering the global correlation and the nonlocal self-similarity (NSS) of RSIs, this paper introduces the FCTN decomposition to the whole RSI and its NSS groups, and proposes a novel nonlocal patch-based FCTN (NL-FCTN) decomposition for RSI inpainting. Different from other nonlocal patch-based methods, the NL-FCTN decomposition-based method, which increases tensor …
School of Geosciences and Info-Physics, Central South University（中南大学地球科学与信息物理学院）
In recent years, deep convolutional neural network (DCNN) has seen a breakthrough progress in natural image recognition because of three points: universal approximation ability via DCNN, large-scale database (such as ImageNet), and supercomputing ability powered by GPU. The remote sensing field is still lacking a large-scale benchmark compared to ImageNet and Place2. In this paper, we propose a remote sensing image classification benchmark (RSI-CB) based on massive, scalable, and diverse crowdsource data. Using crowdsource data, such as Open Street Map (OSM) data, ground objects in remote sens…
