SAM-Assisted Remote Sensing Imagery Semantic Segmentation with Object and Boundary Constraints
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Wuhan University of Science and Technology（武汉科技大学）；The Hong Kong Polytechnic University（香港理工大学）；The University of Hong Kong（香港大学）
Semantic segmentation of remote sensing imagery plays a pivotal role in extracting precise information for diverse down-stream applications. Recent development of the Segment Anything Model (SAM), an advanced general-purpose segmentation model, has revolutionized this field, presenting new avenues for accurate and efficient segmentation. However, SAM is limited to generating segmentation results without class information. Consequently, the utilization of such a powerful general vision model for semantic segmentation in remote sensing images has become a focal point of research. In this paper, we present a streamlined framework aimed at leveraging the raw output of SAM by exploiting two novel concepts called SAM-Generated Object (SGO) and SAM-Generated Boundary (SGB). More specifically, we propose a novel object loss and further introduce a boundary loss as augmentative components to aid in model optimization in a general semantic segmentation framework. Taking into account the content characteristics of SGO, we introduce the concept of object consistency to leverage segmented regions lacking semantic information. By imposing constraints on the consistency of predicted values within objects, the object loss aims to enhance semantic segmentation performance. Furthermore, the boundary loss capitalizes on the distinctive features of SGB by directing the model's attention to the boundary information of the object. Experimental results on two well-known datasets, namely ISPRS Vaihingen and LoveDA Urban, demonstrate the effectiveness of our proposed method. The source code for this work will be accessible at https://github.com/sstary/SSRS.
Thanks to recent advances in CNNs, solid improvements have been made in semantic segmentation of high resolution remote sensing imagery. However, most of the previous works have not fully taken into account the specific difficulties that exist in remote sensing tasks. One of such difficulties is that objects are small and crowded in remote sensing imagery. To tackle with this challenging task we have proposed a novel architecture called local feature extraction (LFE) module attached on top of dilated front-end module. The LFE module is based on our findings that aggressively increasing dilatio…
Recently, the availability of remote sensing imagery from aerial vehicles and satellites constantly improved. For an automated interpretation of such data, deep-learning-based object detectors achieve state-of-the-art performance. However, established object detectors require complete, precise, and correct bounding box annotations for training. In order to create the necessary training annotations for object detectors, imagery can be georeferenced and combined with data from other sources, such as points of interest localized by GPS sensors. Unfortunately, this combination often leads to poor …
Semantic segmentation is a key technique involved in automatic interpretation of high-resolution remote sensing (HRS) imagery and has drawn much attention in the remote sensing community. Deep convolutional neural networks (DCNNs) have been successfully applied to the HRS imagery semantic segmentation task due to their hierarchical representation ability. However, the heavy dependency on a large number of training data with dense annotation and the sensitiveness to the variation of data distribution severely restrict the potential application of DCNNs for the semantic segmentation of HRS image…
With the development of remote sensing technology, the acquisition of remote sensing images is easier and easier, which provides sufficient data resources for the task of detecting remote sensing objects. However, how to detect objects quickly and accurately from many complex optical remote sensing images is a challenging hot issue. In this paper, we propose an efficient anchor free object detector, CenterFPANet. To pursue speed, we use a lightweight backbone and introduce the asymmetric revolution block. To improve the accuracy, we designed the FPA module, which links the feature maps of diff…
New York University（纽约大学）；University of the Philippines Diliman（菲律宾大学迪里曼分校）；University of Washington（华盛顿大学）；German Corporation for International Cooperation GmbH (GIZ)（德国国际合作机构（GIZ））
Rapid damage assessment is of crucial importance to emergency responders during hurricane events, however, the evaluation process is often slow, labor-intensive, costly, and error-prone. New advances in computer vision and remote sensing open possibilities to observe the Earth at a different scale. However, substantial pre-processing work is still required in order to apply state-of-the-art methodology for emergency response. To enable the comparison of methods for automatic detection of damaged buildings from post-hurricane remote sensing imagery taken from both airborne and satellite sensors…
Self-guided Few-shot Semantic Segmentation for Remote Sensing Imagery Based on Large Vision Models
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences（中国科学院大学电子电气与通信工程学院）
Accurate and robust detection of multi-class objects in optical remote sensing images is essential to many real-world applications such as urban planning, traffic control, searching and rescuing, etc. However, state-of-the-art object detection techniques designed for images captured using ground-level sensors usually experience a sharp performance drop when directly applied to remote sensing images, largely due to the object appearance differences in remote sensing images in term of sparse texture, low contrast, arbitrary orientations, large scale variations, etc. This paper presents a novel o…
Beijing Institute of Technology（北京理工大学）；German Aerospace Center (DLR)（德国航空航天中心）；Technical University of Munich (TUM)（慕尼黑工业大学）；Univ. Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；CNRS（法国国家科学研究中心）；Grenoble INP（格勒诺布尔国立高等物理、数学与信息学院）；University of Iceland（冰岛大学）
With the rapid development of spaceborne imaging techniques, object detection in optical remote sensing imagery has drawn much attention in recent decades. While many advanced works have been developed with powerful learning algorithms, the incomplete feature representation still cannot meet the demand for effectively and efficiently handling image deformations, particularly objective scaling and rotation. To this end, we propose a novel object detection framework, called optical remote sensing imagery detector (ORSIm detector), integrating diverse channel features extraction, feature learning…
In this work we consider the application of convolutional neural networks (CNNs) for pixel-wise labeling (a.k.a., semantic segmentation) of remote sensing imagery (e.g., aerial color or hyperspectral imagery). Remote sensing imagery is usually stored in the form of very large images, referred to as "tiles", which are too large to be segmented directly using most CNNs and their associated hardware. As a result, during label inference, smaller sub-images, called "patches", are processed individually and then "stitched" (concatenated) back together to create a tile-sized label map. This approach …
Recent advances in computer vision using deep learning with RGB imagery (e.g., object recognition and detection) have been made possible thanks to the development of large annotated RGB image datasets. In contrast, multispectral image (MSI) and hyperspectral image (HSI) datasets contain far fewer labeled images, in part due to the wide variety of sensors used. These annotations are especially limited for semantic segmentation, or pixel-wise classification, of remote sensing imagery because it is labor intensive to generate image annotations. Low-shot learning algorithms can make effective infe…
For change detection in remote sensing, constructing a training dataset for deep learning models is difficult due to the requirements of bi-temporal supervision. To overcome this issue, single-temporal supervision which treats change labels as the difference of two semantic masks has been proposed. This novel method trains a change detector using two spatially unrelated images with corresponding semantic labels such as building. However, training on unpaired datasets could confuse the change detector in the case of pixels that are labeled unchanged but are visually significantly different. In …
ReCon1M:A Large-scale Benchmark Dataset for Relation Comprehension in Remote Sensing Imagery
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；University of Chinese Academy of Sciences（中国科学院大学）；School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences（中国科学院大学电子电气与通信工程学院）
Carlson Center for Imaging Science, Rochester Institute of Technology（卡尔森成像科学中心，罗切斯特理工学院）
Deep learning continues to push state-of-the-art performance for the semantic segmentation of color (i.e., RGB) imagery; however, the lack of annotated data for many remote sensing sensors (i.e. hyperspectral imagery (HSI)) prevents researchers from taking advantage of this recent success. Since generating sensor specific datasets is time intensive and cost prohibitive, remote sensing researchers have embraced deep unsupervised feature extraction. Although these methods have pushed state-of-the-art performance on current HSI benchmarks, many of these tools are not readily accessible to many re…
MineNetCD: A Benchmark for Global Mining Change Detection on Remote Sensing Imagery
Technical University of Munich（慕尼黑工业大学）；Helmholtz-Zentrum Dresden-Rossendorf (HZDR)（德累斯顿-罗森多夫亥姆霍兹研究中心）；Wuhan University of Science and Technology（武汉科技大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）；Lancaster University（兰卡斯特大学）
German Aerospace Center（德国航空航天中心）；Technical University of Munich（慕尼黑工业大学）
Automatic multi-class object detection in remote sensing images in unconstrained scenarios is of high interest for several applications including traffic monitoring and disaster management. The huge variation in object scale, orientation, category, and complex backgrounds, as well as the different camera sensors pose great challenges for current algorithms. In this work, we propose a new method consisting of a novel joint image cascade and feature pyramid network with multi-size convolution kernels to extract multi-scale strong and weak semantic features. These features are fed into rotation-b…
Hubei University of Technology（湖北工业大学）；Wuhan University（武汉大学）
Land cover classification is a multi-class segmentation task to classify each pixel into a certain natural or man-made category of the earth surface, such as water, soil, natural vegetation, crops, and human infrastructure. Limited by hardware computational resources and memory capacity, most existing studies preprocessed original remote sensing images by down sampling or cropping them into small patches less than 512*512 pixels before sending them to a deep neural network. However, down sampling images incurs spatial detail loss, renders small segments hard to discriminate, and reverses the s…
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；University of Chinese Academy of Sciences（中国科学院大学）；The University of New South Wales（新南威尔士大学）；Australian Defence Force Academy（澳大利亚国防学院）
Semantic segmentation is an essential part of deep learning. In recent years, with the development of remote sensing big data, semantic segmentation has been increasingly used in remote sensing. Deep convolutional neural networks (DCNNs) face the challenge of feature fusion: very-high-resolution remote sensing image multisource data fusion can increase the network's learnable information, which is conducive to correctly classifying target objects by DCNNs; simultaneously, the fusion of high-level abstract features and low-level spatial features can improve the classification accuracy at the bo…
LOGCAN++: Adaptive Local-global class-aware network for semantic segmentation of remote sensing imagery
School of Software Technology, Zhejiang University（浙江大学软件学院）；School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University（上海交通大学电子信息与电气工程学院）；School of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；School of Earth Sciences, Zhejiang University（浙江大学地球科学学院）；Innovation Center of Yangtze River Delta, Zhejiang University（浙江大学长三角创新中心）；University of Exeter（埃克塞特大学）
DeepAndes: A Self-Supervised Vision Foundation Model for Multi-Spectral Remote Sensing Imagery of the Andes
Department of Electrical and Computer Engineering, Vanderbilt University（电气与计算机工程系，范德比尔特大学）；Department of Anthropology, Vanderbilt University（人类学系，范德比尔特大学）；Data Science Institute, Vanderbilt University（数据科学研究院，范德比尔特大学）；Department of Computer Science, Vanderbilt University（计算机科学系，范德比尔特大学）；Department of Mathematics, Vanderbilt University（数学系，范德比尔特大学）；Oak Ridge National Laboratory（橡树岭国家实验室）；Department of Anthropology, Brown University（人类学系，布朗大学）；Department of Radiology, Weill Cornell Medicine（放射学系，韦尔·科恩医学中心）
Intelligence and Space Research Division（情报与空间研究部）；Descartes Labs, Inc.（笛卡尔实验室公司）；Los Alamos National Laboratory（洛斯阿拉莫斯国家实验室）
This paper introduces a new method of generating realistic pervasive changes in the context of evaluating the effectiveness of change detection algorithms in controlled settings. The method, a cycle-consistent adversarial network (CycleGAN), requires low quantities of training data to generate realistic changes. Here we show an application of CycleGAN in creating realistic snow-covered scenes of multispectral Sentinel-2 imagery, and demonstrate how these images can be used as a test bed for anomalous change detection algorithms.
Bridging the Scale Gap: Balanced Tiny and General Object Detection in Remote Sensing Imagery
Key Laboratory of Intelligent Computing & Signal Processing (Anhui University), Ministry of Education（智能计算与信号处理重点实验室（安徽大学））；Anhui Provincial Key Laboratory of Multimodal Cognitive Computation（安徽省多模态认知计算重点实验室）；School of Artificial Intelligence, Anhui University（人工智能学院）；th Research Institute, China Electronics Technology Group Corporation（中国电子科技集团第三十八研究所）；School of Computer Science and Technology, Anhui University（计算机科学与技术学院）
Many significant applications need land cover information of remote sensing images that are acquired from different areas and times, such as change detection and disaster monitoring. However, it is difficult to find a generic land cover classification scheme for different remote sensing images due to the spectral shift caused by diverse acquisition condition. In this paper, we develop a novel land cover classification method that can deal with large-scale data captured from widely distributed areas and different times. Additionally, we establish a large-scale land cover classification dataset …
Chester F. Carlson Center for Imaging Science Rochester Institute of Technology（切斯特·F·卡尔森成像科学中心 罗切斯特理工学院）
Deep convolutional neural networks (DCNNs) have been used to achieve state-of-the-art performance on many computer vision tasks (e.g., object recognition, object detection, semantic segmentation) thanks to a large repository of annotated image data. Large labeled datasets for other sensor modalities, e.g., multispectral imagery (MSI), are not available due to the large cost and manpower required. In this paper, we adapt state-of-the-art DCNN frameworks in computer vision for semantic segmentation for MSI imagery. To overcome label scarcity for MSI data, we substitute real MSI for generated syn…
ChangeDINO: DINOv3-Driven Building Change Detection in Optical Remote Sensing Imagery
1 Inst.\ of Data Science, National Cheng Kung University, \ , Taiwan - 2 Inst.\ of Intelligent Systems College of Artificial Intelligence, \ Yang Ming Chiao Tung University, Hsinchu, Taiwan
In recent years, Fully Convolutional Networks (FCN) has been widely used in various semantic segmentation tasks, including multi-modal remote sensing imagery. How to fuse multi-modal data to improve the segmentation performance has always been a research hotspot. In this paper, a novel end-toend fully convolutional neural network is proposed for semantic segmentation of natural color, infrared imagery and Digital Surface Models (DSM). It is based on a modified DeepUNet and perform the segmentation in a multi-task way. The channels are clustered into groups and processed on different task pipel…
Faculty of Mathematics and Computer Science（数学与计算机科学学院）
In recent years, the geospatial industry has been developing at a steady pace. This growth implies the addition of satellite constellations that produce a copious supply of satellite imagery and other Remote Sensing data on a daily basis. Sometimes, this information, even if in some cases we are referring to publicly available data, it sits unaccounted for due to the sheer size of it. Processing such large amounts of data with the help of human labour or by using traditional automation methods is not always a viable solution from the standpoint of both time and other resources. Within the pres…
Shanghai Jiao Tong University（上海交通大学）；GIPSA-lab, Université Grenoble Alpes, CNRS, Grenoble INP（格勒诺布尔阿尔卑斯大学、法国国家科学研究中心、格勒诺布尔国立理工学院联合GIPSA实验室）；University of Iceland（冰岛大学）；University of Leicester（莱斯特大学）
Detection of objects is extremely important in various aerial vision-based applications. Over the last few years, the methods based on convolution neural networks have made substantial progress. However, because of the large variety of object scales, densities, and arbitrary orientations, the current detectors struggle with the extraction of semantically strong features for small-scale objects by a predefined convolution kernel. To address this problem, we propose the rotation equivariant feature image pyramid network (REFIPN), an image pyramid network based on rotation equivariance convolutio…
Cross-Modal Urban Sensing: Evaluating Sound-Vision Alignment Across Street-Level and Aerial Imagery
Department of Geography, University of South Carolina（南卡罗来纳大学地理系）；Department of Environmental Sciences, Emory University（埃默里大学环境科学系）；University of Canterbury（坎特伯雷大学）；School of Resource and Environmental Sciences, Wuhan University（武汉大学资源与环境科学学院）
School of Automation, Beijing Institute of Technology（北京理工大学自动化学院）
Oriented object detection in remote sensing images has made great progress in recent years. However, most of the current methods only focus on detecting targets, and cannot distinguish fine-grained objects well in complex scenes. In this technical report, we analyzed the key issues of fine-grained object recognition, and use an oriented feature alignment network (OFA-Net) to achieve high-performance fine-grained oriented object recognition in optical remote sensing images. OFA-Net achieves accurate object localization through a rotated bounding boxes refinement module. On this basis, the bound…
Universitat de València（瓦伦西亚大学）；Descartes Lab（笛卡尔实验室）
Anomalous change detection (ACD) is an important problem in remote sensing image processing. Detecting not only pervasive but also anomalous or extreme changes has many applications for which methodologies are available. This paper introduces a nonlinear extension of a full family of anomalous change detectors. In particular, we focus on algorithms that utilize Gaussian and elliptically contoured (EC) distribution and extend them to their nonlinear counterparts based on the theory of reproducing kernels' Hilbert space. We illustrate the performance of the kernel methods introduced in both perv…
In response to the growing importance of geospatial data, its analysis including semantic segmentation becomes an increasingly popular task in computer vision today. Convolutional neural networks are powerful visual models that yield hierarchies of features and practitioners widely use them to process remote sensing data. When performing remote sensing image segmentation, multiple instances of one class with precisely defined boundaries are often the case, and it is crucial to extract those boundaries accurately. The accuracy of segments boundaries delineation influences the quality of the who…
Breaking the Box: Enhancing Remote Sensing Image Segmentation with Freehand Sketches
Huzhou University（湖州大学）；Singapore University of Technology and Design（新加坡科技设计大学）；University of Science and Technology of China（中国科学技术大学）；Zhejiang University（浙江大学）；KOKONI 3D, Moxin (Huzhou) Tech. Co., LTD.（墨芯（湖州）科技有限公司（KOKONI 3D））
School of Remote Sensing and Information Engineering, Wuhan University（武汉大学遥感信息工程学院）
As one kind of architecture from the deep learning family, deep semantic segmentation network (DSSN) achieves a certain degree of success on the semantic segmentation task and obviously outperforms the traditional methods based on hand-crafted features. As a classic data-driven technique, DSSN can be trained by an end-to-end mechanism and competent for employing the low-level and mid-level cues (i.e., the discriminative image structure) to understand images, but lacks the high-level inference ability. By contrast, human beings have an excellent inference capacity and can be able to reliably in…
University of Shanghai for Science and Technology（上海理工大学）；Indiana University Bloomington（印第安纳大学伯明顿分校）；Tianjin University（天津大学）；Suffolk University（萨福克大学）
Vegetation is the natural linkage connecting soil, atmosphere and water. It can represent the change of land cover to a certain extent and serve as an indicator for global change research. Methods for measuring coverage can be divided into two types: surface measurement and remote sensing. Because vegetation cover has significant spatial and temporal differentiation characteristics, remote sensing has become an important technical means to estimate vegetation coverage. This paper firstly uses U-net to perform remote sensing image semantic segmentation training, then uses the result of semantic…
With the acceleration of the urban expansion, urban change detection (UCD), as a significant and effective approach, can provide the change information with respect to geospatial objects for dynamical urban analysis. However, existing datasets suffer from three bottlenecks: (1) lack of high spatial resolution images; (2) lack of semantic annotation; (3) lack of long-range multi-temporal images. In this paper, we propose a large scale benchmark dataset, termed Hi-UCD. This dataset uses aerial images with a spatial resolution of 0.1 m provided by the Estonia Land Board, including three-time phas…
School of Aerospace Engineering, Tsinghua University（清华大学航天航空学院）
Cross-modality fusing complementary information of multispectral remote sensing image pairs can improve the perception ability of detection algorithms, making them more robust and reliable for a wider range of applications, such as nighttime detection. Compared with prior methods, we think different features should be processed specifically, the modality-specific features should be retained and enhanced, while the modality-shared features should be cherry-picked from the RGB and thermal IR modalities. Following this idea, a novel and lightweight multispectral feature fusion approach with joint…
MKSNet: Advanced Small Object Detection in Remote Sensing Imagery with Multi-Kernel and Dual Attention Mechanisms
School of Computer Science, Beijing Institute of Technology, Beijing, China（计算机科学学院，北京理工大学，北京）；Tangshan Research Institute, Beijing Institute of Technology, Tangshan, China（唐山研究所，北京理工大学，唐山）；Shanghai Institute of Satellite Engineering, Shanghai, China（上海卫星工程研究所，上海）
