About half of all optical observations collected via spaceborne satellites are affected by haze or clouds. Consequently, cloud coverage affects the remote sensing practitioner's capabilities of a continuous and seamless monitoring of our planet. This work addresses the challenge of optical satellite image reconstruction and cloud removal by proposing a novel multi-modal and multi-temporal data set called SEN12MS-CR-TS. We propose two models highlighting the benefits and use cases of SEN12MS-CR-TS: First, a multi-modal multi-temporal 3D-Convolution Neural Network that predicts a cloud-free image from a sequence of cloudy optical and radar images. Second, a sequence-to-sequence translation model that predicts a cloud-free time series from a cloud-covered time series. Both approaches are evaluated experimentally, with their respective models trained and tested on SEN12MS-CR-TS. The conducted experiments highlight the contribution of our data set to the remote sensing community as well as the benefits of multi-modal and multi-temporal information to reconstruct noisy information. Our data set is available at https://patrickTUM.github.io/cloud_removal
Optical remote sensing imagery has been widely used in many fields due to its high resolution and stable geometric properties. However, remote sensing imagery is inevitably affected by climate, especially clouds. Removing the cloud in the high-resolution remote sensing satellite image is an indispensable pre-processing step before analyzing it. For the sake of large-scale training data, neural networks have been successful in many image processing tasks, but the use of neural networks to remove cloud in remote sensing imagery is still relatively small. We adopt generative adversarial network t…
IDF-CR: Iterative Diffusion Process for Divide-and-Conquer Cloud Removal in Remote-sensing Images
School of Information Engineering, Guangdong University of Technology（广东工业大学信息工程学院）；CRRC Academy Co., Ltd.（中车研究院有限公司）；School of Computer Science, Sun Yat-sen University（中山大学计算机学院）
Satellite images are often contaminated by clouds. Cloud removal has received much attention due to the wide range of satellite image applications. As the clouds thicken, the process of removing the clouds becomes more challenging. In such cases, using auxiliary images such as near-infrared or synthetic aperture radar (SAR) for reconstructing is common. In this study, we attempt to solve the problem using two generative adversarial networks (GANs). The first translates SAR images into optical images, and the second removes clouds using the translated images of prior GAN. Also, we propose dilat…
Institute of Electronics, Chinese Academy of Sciences（中国科学院电子学研究所）
Cloud-based overlays are often present in optical remote sensing images, thus limiting the application of acquired data. Removing clouds is an indispensable pre-processing step in remote sensing image analysis. Deep learning has achieved great success in the field of remote sensing in recent years, including scene classification and change detection. However, deep learning is rarely applied in remote sensing image removal clouds. The reason is the lack of data sets for training neural networks. In order to solve this problem, this paper first proposed the Remote sensing Image Cloud rEmoving da…
In remote sensing images, the presence of thick cloud accompanying cloud shadow is a high probability event, which can affect the quality of subsequent processing and limit the scenarios of application. Hence, removing the thick cloud and cloud shadow as well as recovering the cloud-contaminated pixels is indispensable to make good use of remote sensing images. In this paper, a novel thick cloud removal method for remote sensing images based on temporal smoothness and sparsity-regularized tensor optimization (TSSTO) is proposed. The basic idea of TSSTO is that the thick cloud and cloud shadow …
SADER: Structure-Aware Diffusion Framework with DEterministic Resampling for Multi-Temporal Remote Sensing Cloud Removal
College of Literature, Science, and the Arts, University of Michigan（文学、科学与艺术学院，密歇根大学）；School of Computer Science and Technology, Tongji University（计算机科学与技术学院，同济大学）
This paper attempts to undertake the study of three types of noise such as Salt and Pepper (SPN), Random variation Impulse Noise (RVIN), Speckle (SPKN). Different noise densities have been removed between 10% to 60% by using five types of filters as Mean Filter (MF), Adaptive Wiener Filter (AWF), Gaussian Filter (GF), Standard Median Filter (SMF) and Adaptive Median Filter (AMF). The same is applied to the Saturn remote sensing image and they are compared with one another. The comparative study is conducted with the help of Mean Square Errors (MSE) and Peak-Signal to Noise Ratio (PSNR). So as …
Destriping is a classical problem in remote sensing image processing. Although considerable effort has been made to remove stripes, few of the existing methods can eliminate stripe noise with arbitrary orientations. This situation makes the removal of oblique stripes in the higher-level remote sensing products become an unfinished and urgent issue. To overcome the challenging problem, we propose a novel destriping model which is self-adjusted to different orientations of stripe noise. First of all, the oriented variation model is designed to accomplish the stripe orientation approximation. In …
Centre for Research-based Innovation Visual Intelligence（可视化研究型创新中心）
Multi-modality data is becoming readily available in remote sensing (RS) and can provide complementary information about the Earth's surface. Effective fusion of multi-modal information is thus important for various applications in RS, but also very challenging due to large domain differences, noise, and redundancies. There is a lack of effective and scalable fusion techniques for bridging multiple modality encoders and fully exploiting complementary information. To this end, we propose a new multi-modality network (MultiModNet) for land cover mapping of multi-modal remote sensing data based o…
The State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University（武汉大学测绘遥感信息工程国家重点实验室）；School of Remote Sensing and Information Engineering, Wuhan University（武汉大学遥感信息工程学院）
In remote sensing images, the existence of the thin cloud is an inevitable and ubiquitous phenomenon that crucially reduces the quality of imageries and limits the scenarios of application. Therefore, thin cloud removal is an indispensable procedure to enhance the utilization of remote sensing images. Generally, even though contaminated by thin clouds, the pixels still retain more or less surface information. Hence, different from thick cloud removal, thin cloud removal algorithms normally concentrate on inhibiting the cloud influence rather than substituting the cloud-contaminated pixels. Mea…
Remote sensing images often suffer from cloud cover. Cloud removal is required in many applications of remote sensing images. Multitemporal-based methods are popular and effective to cope with thick clouds. This paper contributes to a summarization and experimental comparation of the existing multitemporal-based methods. Furthermore, we propose a spatiotemporal-fusion with poisson-adjustment method to fuse multi-sensor and multi-temporal images for cloud removal. The experimental results show that the proposed method has potential to address the problem of accuracy reduction of cloud removal i…
Diffusion Enhancement for Cloud Removal in Ultra-Resolution Remote Sensing Imagery
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Peking University（北京大学）；Peng Cheng Laboratory（鹏城实验室）；Wuhan University of Science and Technology（武汉科技大学）
School of Aerospace Engineering, Tsinghua University（清华大学航天航空学院）
Cross-modality fusing complementary information of multispectral remote sensing image pairs can improve the perception ability of detection algorithms, making them more robust and reliable for a wider range of applications, such as nighttime detection. Compared with prior methods, we think different features should be processed specifically, the modality-specific features should be retained and enhanced, while the modality-shared features should be cherry-picked from the RGB and thermal IR modalities. Following this idea, a novel and lightweight multispectral feature fusion approach with joint…
Indian Institute of Technology Bombay（印度理工学院孟买分校）；German Aerospace Center (DLR)（德国航空航天中心）
We address the problem of cross-modal information retrieval in the domain of remote sensing. In particular, we are interested in two application scenarios: i) cross-modal retrieval between panchromatic (PAN) and multi-spectral imagery, and ii) multi-label image retrieval between very high resolution (VHR) images and speech based label annotations. Notice that these multi-modal retrieval scenarios are more challenging than the traditional uni-modal retrieval approaches given the inherent differences in distributions between the modalities. However, with the growing availability of multi-source …
Task-Guided Prompting for Unified Remote Sensing Image Restoration
School of Electronic and Information Engineering, Ningbo University of Technology（宁波工程学院电子与信息工程学院）；University of Exeter（埃克塞特大学）；Engineering Research Center of Intelligent Finance, Ministry of Education（教育部智能金融工程研究中心）；School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics（西南财经大学计算机与人工智能学院）
Centre of Studies in Resources Engineering (CSRE), Indian Institute of Technology Bombay（印度理工学院孟买分校资源工程研究中心）；German Aerospace Center (DLR)（德国航空航天中心）
Conventional existing retrieval methods in remote sensing (RS) are often based on a uni-modal data retrieval framework. In this work, we propose a novel inter-modal triplet-based zero-shot retrieval scheme utilizing a sketch-based representation of RS data. The proposed scheme performs efficiently even when the sketch representations are marginally prototypical of the image. We conducted experiments on a new bi-modal image-sketch dataset called Earth on Canvas (EoC) conceived during this study. We perform a thorough bench-marking of this dataset and demonstrate that the proposed network outper…
DeepLight: Reconstructing High-Resolution Observations of Nighttime Light With Multi-Modal Remote Sensing Data
Tsinghua University - Xi’an Institute of Surveying and Mapping Joint Research Center（清华大学-西安测绘联合研究中心）；National supercomputing center in Shenzhen, Tsinghua University（清华大学深圳国家超级计算中心）；The University of Hong Kong（香港大学）；Sun-Yat San University（中山大学）
German Aerospace Center（德国航空航天中心）；Technical University of Munich（慕尼黑工业大学）；The University of Tokyo（东京大学）；RIKEN（日本理化学研究所）；Wuhan University（武汉大学）；Univ. Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；INRIA（法国国家信息与自动化研究所）；CNRS（法国国家科学研究中心）；Grenoble INP（格勒诺布尔理工学院）；Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）
This paper addresses the problem of semi-supervised transfer learning with limited cross-modality data in remote sensing. A large amount of multi-modal earth observation images, such as multispectral imagery (MSI) or synthetic aperture radar (SAR) data, are openly available on a global scale, enabling parsing global urban scenes through remote sensing imagery. However, their ability in identifying materials (pixel-wise classification) remains limited, due to the noisy collection environment and poor discriminative information as well as limited number of well-annotated training images. To this…
RemoteDet-Mamba: A Hybrid Mamba-CNN Network for Multi-modal Object Detection in Remote Sensing Images
School of Computer Science, Beijing University of Posts and Telecommunications, Beijing, China（计算机科学学院，北京邮电大学）；School of Electronic Engineering, Beijing University of Posts and Telecommunications, Beijing, China（电子工程学院，北京邮电大学）
UniDiff: Parameter-Efficient Adaptation of Diffusion Models for Land Cover Classification with Multi-Modal Remotely Sensed Imagery and Sparse Annotations
Multi-modal Uncertainty Robust Tree Cover Segmentation For High-Resolution Remote Sensing Images
School of Information and Electronics, Beijing Institute of Technology（信息与电子学院，北京理工大学）；National Key Laboratory of Science and Technology on Space-Born Intelligent Information Processing（空间智能信息处理国家重点实验室）；Department of Electrical and Computer Engineering, University of Delaware（电气与计算机工程系，德雷塞尔大学）；Swiss Federal Institute for Forest, Snow, and Landscape Research WSL（瑞士森林、雪和景观研究联邦 institute WSL）
SARU: A Shadow-Aware and Removal Unified Framework for Remote Sensing Images with New Benchmarks
MOE Key Lab of ICSP, Anhui Provincial Key Lab of Multimodal Cognitive Computation, IMIS Lab of Anhui Province, School of Computer Science and Technology, Anhui University, Hefei, China（教育部信息与通信系统重点实验室、安徽省多模态认知计算重点实验室、安徽省IMIS实验室、计算机科学与技术学院、安徽大学、合肥，中国）；Graduate School of Frontier Sciences, The University of Tokyo, Chiba, 277-8561, Japan（前沿科学研究生院、东京大学、千叶，日本）
University of Missouri（密苏里大学）；University of Michigan（密歇根大学）；University of Florida（佛罗里达大学）
In remote sensing, each sensor can provide complementary or reinforcing information. It is valuable to fuse outputs from multiple sensors to boost overall performance. Previous supervised fusion methods often require accurate labels for each pixel in the training data. However, in many remote sensing applications, pixel-level labels are difficult or infeasible to obtain. In addition, outputs from multiple sensors often have different resolution or modalities. For example, rasterized hyperspectral imagery presents data in a pixel grid while airborne Light Detection and Ranging (LiDAR) generates…
The implementation of automatic image registration is still difficult in various applications. In this paper, an automatic image registration approach through line-support region segmentation and geometrical outlier removal (ALRS-GOR) is proposed. This new approach is designed to address the problems associated with the registration of images with affine deformations and inconsistent content, such as remote sensing images with different spectral content or noise interference, or map images with inconsistent annotations. To begin with, line-support regions, namely a straight region whose points…
Asynchronous Remote Sensing Time-Series Fusion for Cloud Removal and Anytime Reconstruction
School of Computing and Augmented Intelligence, Arizona State University（计算与增强智能学院，亚利桑那州立大学）；School of Geographical Sciences and Urban Planning, Arizona State University（地理科学与城市规划学院，亚利桑那州立大学）；Woodwell Climate Research Center（伍德沃德气候研究中心）
CR-JEPA: Cross-Modal Joint-Embedding Predictive Learning for Remote Sensing Image Retrieval
Space Applications Centre, Indian Space Research Organisation（印度空间研究组织空间应用中心）；Centre of Studies in Resources Engineering, Indian Institute of Technology Bombay（印度理工学院孟买资源工程研究中心）
The development of accurate and scalable cross-modal image-text retrieval methods, where queries from one modality (e.g., text) can be matched to archive entries from another (e.g., remote sensing image) has attracted great attention in remote sensing (RS). Most of the existing methods assume that a reliable multi-modal training set with accurately matched text-image pairs is existing. However, this assumption may not always hold since the multi-modal training sets may include noisy pairs (i.e., textual descriptions/captions associated to training images can be noisy), distorting the learning …
Space Applications Centre, Indian Space Research Organisation（印度空间研究组织空间应用中心）；Centre of Studies in Resources Engineering, Indian Institute of Technology Bombay（印度理工学院孟买资源工程研究中心）
Galileo: Learning Global & Local Features of Many Remote Sensing Modalities
Mila -- Quebec AI Institute（魁北克人工智能研究所）；McGill University（麦吉尔大学）；Arizona State University（亚利桑那州立大学）；Carleton University（卡尔顿大学）；Allen Institute for AI (Ai2)（人工智能研究所）；University of British Columbia（不列颠哥伦比亚大学）；Vector Institute（向量研究所）
The development of cross-modal retrieval systems that can search and retrieve semantically relevant data across different modalities based on a query in any modality has attracted great attention in remote sensing (RS). In this paper, we focus our attention on cross-modal text-image retrieval, where queries from one modality (e.g., text) can be matched to archive entries from another (e.g., image). Most of the existing cross-modal text-image retrieval systems in RS require a high number of labeled training samples and also do not allow fast and memory-efficient retrieval. These issues limit th…
State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University（测绘遥感信息工程国家重点实验室，武汉大学）；Electronic Information School, Wuhan University（电子信息学院，武汉大学）
Due to the availability of multi-modal remote sensing (RS) image archives, one of the most important research topics is the development of cross-modal RS image retrieval (CM-RSIR) methods that search semantically similar images across different modalities. Existing CM-RSIR methods require the availability of a high quality and quantity of annotated training images. The collection of a sufficient number of reliable labeled images is time consuming, complex and costly in operational scenarios, and can significantly affect the final accuracy of CM-RSIR. In this paper, we introduce a novel self-su…
SGMA: Semantic-Guided Modality-Aware Segmentation for Remote Sensing with Incomplete Multimodal Data
State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University（武汉大学测绘遥感信息工程国家重点实验室）；Hangzhou Institute of Technology, Xidian University（西安电子科技大学杭州研究院）；School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院）
Transformer-based Multi-Modal Learning for Multi Label Remote Sensing Image Classification
Faculty of Electrical Engineering and Computer Science, Technische Universität Berlin（柏林工业大学电机工程与计算机科学学院）；BIFOLD – Berlin Institute for the Foundations of Learning and Data（柏林学习与数据基础研究所）
Prototype-Based Information Compensation Network for Multi-Source Remote Sensing Data Classification
State Key Laboratory of Physical Oceanography, Ocean University of China（海洋大学物理海洋国家重点实验室）；Department of Electrical and Computer Engineering, Mississippi State University（密苏里州立大学电气与计算机工程系）
Any2Any: Unified Arbitrary Modality Translation for Remote Sensing
National Engineering Research Center for Multimedia Software, Institute of Artificial Intelligence, School of Computer Science, Wuhan University（多媒体软件国家工程研究中心、人工智能研究院、计算机科学学院、武汉大学）；Hubei Key Laboratory of Multimedia（湖北省多媒体重点实验室）；Zhongguancun Academy, Beijing, China. 100094（中关村学院，北京，中国。100094）；School of Electronic Information, Wuhan University, Wuhan, China（电子信息学院，武汉大学，武汉，中国）；School of Automation, Beijing Institute of Technology（自动化学院，北京理工大学）；State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan, China（测绘、制图与遥感信息工程国家重点实验室，武汉大学，武汉，中国）
A Unified Foundation Model for All-in-One Multi-Modal Remote Sensing Image Restoration and Fusion with Language Prompting
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences（中国科学院大学电子电气与通信工程学院）
