Remote sensing image fusion technology (pan-sharpening) is an important means to improve the information capacity of remote sensing images. Inspired by the efficient arameter space posteriori sampling of Bayesian neural networks, in this paper we propose a Bayesian Generative Adversarial Network based on Preconditioned Stochastic Gradient Langevin Dynamics (PGSLD-BGAN) to improve pan-sharpening tasks. Unlike many traditional generative models that consider only one optimal solution (might be locally optimal), the proposed PGSLD-BGAN performs Bayesian inference on the network parameters, and explore the generator posteriori distribution, which assists selecting the appropriate generator parameters. First, we build a two-stream generator network with PAN and MS images as input, which consists of three parts: feature extraction, feature fusion and image reconstruction. Then, we leverage Markov discriminator to enhance the ability of generator to reconstruct the fusion image, so that the result image can retain more details. Finally, introducing Preconditioned Stochastic Gradient Langevin Dynamics policy, we perform Bayesian inference on the generator network. Experiments on QuickBird and WorldView datasets show that the model proposed in this paper can effectively fuse PAN and MS images, and be competitive with even superior to state of the arts in terms of subjective and objective metrics.
Several remote sensing software packages are used to the explicit purpose of analyzing and visualizing remotely sensed data, with the developing of remote sensing sensor technologies from last ten years. Accord-ing to literature, the remote sensing is still the lack of software tools for effective information extraction from remote sensing data. So, this paper provides a state-of-art of multi-sensor image fusion technologies as well as review on the quality evaluation of the single image or fused images in the commercial remote sensing pack-ages. It also introduces program (ALwassaiProcess) de…
Scientific Research Group in Egypt (SRGE)（埃及科学研究组）；Benha University（本哈大学）；Damietta University（达米埃塔大学）；Cairo University（开罗大学）
Remote sensing image fusion is an effective way to use a large volume of data from multisensor images. Most earth satellites such as SPOT, Landsat 7, IKONOS and QuickBird provide both panchromatic (Pan) images at a higher spatial resolution and multispectral (MS) images at a lower spatial resolution and many remote sensing applications require both high spatial and high spectral resolutions, especially for GIS based applications. An effective image fusion technique can produce such remotely sensed images. Image fusion is the combination of two or more different images to form a new image by us…
Multi-Focus Image Fusion (MFIF) is a promising image enhancement technique to obtain all-in-focus images meeting visual needs and it is a precondition of other computer vision tasks. One of the research trends of MFIF is to avoid the defocus spread effect (DSE) around the focus/defocus boundary (FDB). In this paper,we propose a network termed MFIF-GAN to attenuate the DSE by generating focus maps in which the foreground region are correctly larger than the corresponding objects. The Squeeze and Excitation Residual module is employed in the network. By combining the prior knowledge of training …
Remote sensing image fusion (also known as pan-sharpening) aims at generating high resolution multi-spectral (MS) image from inputs of a high spatial resolution single band panchromatic (PAN) image and a low spatial resolution multi-spectral image. Inspired by the astounding achievements of convolutional neural networks (CNNs) in a variety of computer vision tasks, in this paper, we propose a two-stream fusion network (TFNet) to address the problem of pan-sharpening. Unlike previous CNN based methods that consider pan-sharpening as a super resolution problem and perform pan-sharpening in pixel…
A Unified Foundation Model for All-in-One Multi-Modal Remote Sensing Image Restoration and Fusion with Language Prompting
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences（中国科学院大学电子电气与通信工程学院）
Computer Network Information Center, Chinese Academy of Sciences（中国科学院计算机网络信息中心）；University of Chinese Academy of Sciences（中国科学院大学）；Quaid-e-Awam University Engineering Science and Technology（奎德阿瓦姆工程科技大学）；Salim Habib University（萨利姆·哈比卜大学）；Waterford Institute of Technology（沃特福德理工学院）
Single Image Super-resolution (SISR) produces high-resolution images with fine spatial resolutions from aremotely sensed image with low spatial resolution. Recently, deep learning and generative adversarial networks(GANs) have made breakthroughs for the challenging task of single image super-resolution (SISR). However, thegenerated image still suffers from undesirable artifacts such as, the absence of texture-feature representationand high-frequency information. We propose a frequency domain-based spatio-temporal remote sensingsingle image super-resolution technique to reconstruct the HR image…
National University of Singapore（新加坡国立大学）；Wuhan University（武汉大学）；Chinese University of Hong Kong（香港中文大学）
Color correction for underwater images has received increasing interests, due to its critical role in facilitating available mature vision algorithms for underwater scenarios. Inspired by the stunning success of deep convolutional neural networks (DCNNs) techniques in many vision tasks, especially the strength in extracting features in multiple scales, we propose a deep multi-scale feature fusion net based on the conditional generative adversarial network (GAN) for underwater image color correction. In our network, multi-scale features are extracted first, followed by augmenting local features…
FusionMamba: Efficient Remote Sensing Image Fusion with State Space Model
Institute of Automation, Chinese Academy of Sciences (CASIA)（中国科学院自动化研究所）；University of Chinese Academy of Sciences (UCAS)（中国科学院大学）；University of Electronic Science and Technology of China (UESTC)（电子科技大学）；Hong Kong Institute of Science and Innovation, Chinese Academy of Sciences（中国科学院香港创新研究院）
Deep learning is a rapidly developing approach in the field of infrared and visible image fusion. In this context, the use of dense blocks in deep networks significantly improves the utilization of shallow information, and the combination of the Generative Adversarial Network (GAN) also improves the fusion performance of two source images. We propose a new method based on dense blocks and GANs , and we directly insert the input image-visible light image in each layer of the entire network. We use SSIM and gradient loss functions that are more consistent with perception instead of mean square e…
An Integrated Framework for the Heterogeneous Spatio-Spectral-Temporal Fusion of Remote Sensing Images
School of Resource and Environmental Science, Wuhan University（武汉大学资源与环境科学学院）；School of Geodesy and Geomatics, Wuhan University（武汉大学测绘学院）；Collaborative Innovation Center of Geospatial Technology, Wuhan University（武汉大学地球空间信息技术协同创新中心）；State Key Laboratory of Information Engineering, Survey Mapping and Remote Sensing, Wuhan University（武汉大学测绘遥感信息工程国家重点实验室）
Yeshwant Mahavidyala College (SRTMU)（耶什万特马哈维迪亚拉学院（SRTMU））
Remote sensing has proven to be a powerful tool for the monitoring of the Earth surface to improve our perception of our surroundings has led to unprecedented developments in sensor and information technologies. However, technologies for effective use of the data and for extracting useful information from the data of Remote sensing are still very limited since no single sensor combines the optimal spectral, spatial and temporal resolution. This paper briefly reviews the limitations of satellite remote sensing. Also, reviews on the problems of image fusion techniques. The conclusion of this, Ac…
Adaptive 3D Convolution for Remote Sensing Image Fusion
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（多模态人工智能系统国家重点实验室，自动化研究所，中国科学院）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；School of Mathematical Sciences/Multi-Hazard Early Warning Key Laboratory of Sichuan Province, University of Electronic Science and Technology of China（数学科学学院/四川省多灾种早期预警重点实验室，电子科技大学）；Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science and Innovation, Chinese Academy of Sciences（人工智能与机器人中心，香港科学院，中国科学院）
Shiraz University of Technology（设拉子理工大学）；Persian Gulf University（波斯湾大学）；Yazd Branch, Islamic Azad University（伊斯兰阿扎德大学亚兹德分校）；Najafabad Branch, Islamic Azad University（伊斯兰阿扎德大学纳杰法巴德分校）
In interpretation of remote sensing images, it is possible that some images which are supplied by different sensors become incomprehensible. For better visual perception of these images, it is essential to operate series of pre-processing and elementary corrections and then operate a series of main processing steps for more precise analysis on the images. There are several approaches for processing which are depended on the type of remote sensing images. The discussed approach in this article, i.e. image fusion, is the use of natural colors of an optical image for adding color to a grayscale s…
College of Computer Science & Technology, Qingdao University（青岛大学计算机科学与技术学院）；School of Computer Science & Engineering, South China University of Technology（华南理工大学计算机科学与工程学院）
University of Cape Town（开普敦大学）；University of Lagos（拉各斯大学）
The remote sensing community has identified data fusion as one of the key challenging topics of the 21st century. The subject of image fusion in two-dimensional (2D) space has been covered in several published reviews. However, the special case of 2.5D/3D Digital Elevation Model (DEM) fusion has not been addressed till date. DEM fusion is a key application of data fusion in remote sensing. It takes advantage of the complementary characteristics of multi-source DEMs to deliver a more complete, accurate and reliable elevation dataset. Although several methods for fusing DEMs have been developed,…
Remote sensing images (RSIs) in real scenes may be disturbed by multiple factors such as optical blur, undersampling, and additional noise, resulting in complex and diverse degradation models. At present, the mainstream SR algorithms only consider a single and fixed degradation (such as bicubic interpolation) and cannot flexibly handle complex degradations in real scenes. Therefore, designing a super-resolution (SR) model that can cope with various degradations is gradually attracting the attention of researchers. Some studies first estimate the degradation kernels and then perform degradation…
Key Laboratory of Technology in Geo-spatial Information Processing and Application System, Chinese Academy of Sciences（中国科学院空间信息处理与应用系统技术重点实验室）
With the development of deep learning, supervised learning has frequently been adopted to classify remotely sensed images using convolutional networks (CNNs). However, due to the limited amount of labeled data available, supervised learning is often difficult to carry out. Therefore, we proposed an unsupervised model called multiple-layer feature-matching generative adversarial networks (MARTA GANs) to learn a representation using only unlabeled data. MARTA GANs consists of both a generative model $G$ and a discriminative model $D$. We treat $D$ as a feature extractor. To fit the complex prope…
Remote sensing images and techniques are powerful tools to investigate earth surface. Data quality is the key to enhance remote sensing applications and obtaining a clear and noise-free set of data is very difficult in most situations due to the varying acquisition (e.g., atmosphere and season), sensor, and platform (e.g., satellite angles and sensor characteristics) conditions. With the increasing development of satellites, nowadays Terabytes of remote sensing images can be acquired every day. Therefore, information and data fusion can be particularly important in the remote sensing community…
Nanyang Technological University（南洋理工大学）；Institute for Infocomm, A*STAR（新加坡科技研究局资讯通信研究院）
Recent advances in generative adversarial networks (GANs) have shown great potentials in realistic image synthesis whereas most existing works address synthesis realism in either appearance space or geometry space but few in both. This paper presents an innovative Spatial Fusion GAN (SF-GAN) that combines a geometry synthesizer and an appearance synthesizer to achieve synthesis realism in both geometry and appearance spaces. The geometry synthesizer learns contextual geometries of background images and transforms and places foreground objects into the background images unanimously. The appeara…
University of Missouri（密苏里大学）；University of Florida（佛罗里达大学）
In classifier (or regression) fusion the aim is to combine the outputs of several algorithms to boost overall performance. Standard supervised fusion algorithms often require accurate and precise training labels. However, accurate labels may be difficult to obtain in many remote sensing applications. This paper proposes novel classification and regression fusion models that can be trained given ambiguosly and imprecisely labeled training data in which training labels are associated with sets of data points (i.e., "bags") instead of individual data points (i.e., "instances") following a multipl…
Institute of Remote Sensing and Digital Earth, Chinese Academy of Sciences（中国科学院遥感与数字地球研究所）；University of Chinese Academy of Sciences（中国科学院大学）
Lack of annotated samples greatly restrains the direct application of deep learning in remote sensing image scene classification. Although researches have been done to tackle this issue by data augmentation with various image transformation operations, they are still limited in quantity and diversity. Recently, the advent of the unsupervised learning based generative adversarial networks (GANs) bring us a new way to generate augmented samples. However, such GAN-generated samples are currently only served for training GANs model itself and for improving the performance of the discriminator in G…
Indian Institute of Technology Bombay（印度理工学院孟买分校）；Purdue University（普渡大学）
We tackle the problem of image inpainting in the remote sensing domain. Remote sensing images possess high resolution and geographical variations, that render the conventional inpainting methods less effective. This further entails the requirement of models with high complexity to sufficiently capture the spectral, spatial and textural nuances within an image, emerging from its high spatial variability. To this end, we propose a novel inpainting method that individually focuses on each aspect of an image such as edges, colour and texture using a task specific GAN. Moreover, each individual GAN…
University of Chinese Academy of Sciences（中国科学院大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；ByteDance Ltd（字节跳动有限公司）；Nanjing University（南京大学）
Exemplar-based image translation refers to the task of generating images with the desired style, while conditioning on certain input image. Most of the current methods learn the correspondence between two input domains and lack the mining of information within the domains. In this paper, we propose a more general learning approach by considering two domain features as a whole and learning both inter-domain correspondence and intra-domain potential information interactions. Specifically, we propose a Cross-domain Feature Fusion Transformer (CFFT) to learn inter- and intra-domain feature fusion.…
University of Missouri（密苏里大学）；University of Michigan（密歇根大学）；University of Florida（佛罗里达大学）
In remote sensing, each sensor can provide complementary or reinforcing information. It is valuable to fuse outputs from multiple sensors to boost overall performance. Previous supervised fusion methods often require accurate labels for each pixel in the training data. However, in many remote sensing applications, pixel-level labels are difficult or infeasible to obtain. In addition, outputs from multiple sensors often have different resolution or modalities. For example, rasterized hyperspectral imagery presents data in a pixel grid while airborne Light Detection and Ranging (LiDAR) generates…
We look into Generative Adversarial Network (GAN), its prevalent variants and applications in a number of sectors. GANs combine two neural networks that compete against one another using zero-sum game theory, allowing them to create much crisper and discrete outputs. GANs can be used to perform image processing, video generation and prediction, among other computer vision applications. GANs can also be utilised for a variety of science-related activities, including protein engineering, astronomical data processing, remote sensing image dehazing, and crystal structure synthesis. Other notable f…
China University of Geosciences（中国地质大学）；Beijing Normal University（北京师范大学）
With a small number of labeled samples for training, it can save considerable manpower and material resources, especially when the amount of high spatial resolution remote sensing images (HSR-RSIs) increases considerably. However, many deep models face the problem of overfitting when using a small number of labeled samples. This might degrade HSRRSI retrieval accuracy. Aiming at obtaining more accurate HSR-RSI retrieval performance with small training samples, we develop a deep metric learning approach with generative adversarial network regularization (DML-GANR) for HSR-RSI retrieval. The DML…
University of Alberta（阿尔伯塔大学）；Ludwig-Maximilians-Universität München（慕尼黑路德维希-马克西米利安大学（慕尼黑大学））；Alberta Geological Survey（阿尔伯塔地质调查局）；Alberta Energy Regulator（阿尔伯塔能源监管局）
The detection performance of small objects in remote sensing images is not satisfactory compared to large objects, especially in low-resolution and noisy images. A generative adversarial network (GAN)-based model called enhanced super-resolution GAN (ESRGAN) shows remarkable image enhancement performance, but reconstructed images miss high-frequency edge information. Therefore, object detection performance degrades for small objects on recovered noisy and low-resolution remote sensing images. Inspired by the success of edge enhanced GAN (EEGAN) and ESRGAN, we apply a new edge-enhanced super-re…
Shenzhen Institutes of Advanced Technology, CAS（中国科学院深圳先进技术研究院）；Beijing Futong Dongfang Technology Co, Ltd（北京富通东方科技有限公司）；Beijing University of Chemical Technology（北京化工大学）
Robust semantic segmentation of VHR remote sensing images from UAV sensors is critical for earth observation, land use, land cover or mapping applications. Several factors such as shadows, weather disruption and camera shakes making this problem highly challenging, especially only using RGB images. In this paper, we propose the use of multi-modality data including NIR, RGB and DSM to increase robustness of segmentation in blurred or partially damaged VHR remote sensing images. By proposing a cascaded dense encoder-decoder network and the SELayer based fusion and assembling techniques, the prop…
In recent years, Fully Convolutional Networks (FCN) has been widely used in various semantic segmentation tasks, including multi-modal remote sensing imagery. How to fuse multi-modal data to improve the segmentation performance has always been a research hotspot. In this paper, a novel end-toend fully convolutional neural network is proposed for semantic segmentation of natural color, infrared imagery and Digital Surface Models (DSM). It is based on a modified DeepUNet and perform the segmentation in a multi-task way. The channels are clustered into groups and processed on different task pipel…
Manifold Constraint Regularization for Remote Sensing Image Generation
Institute of Software Chinese Academy of Sciences, University of Chinese Academy of Sciences（中国科学院大学中国科学院软件研究所）；Tsinghua University（清华大学）；Hong Kong University of Science and Technology（香港科技大学）
本文首次提出流形约束正则化方法，通过新度量评估数据流形结构并约束 GAN 训练，缓解遥感图像生成中的过拟合问题，同时提升生成质量与下游任务性能。
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Adobe Inc.（奥多比公司）
Recently, convolutional neural networks (CNNs) have achieved great improvements in single image dehazing and attained much attention in research. Most existing learning-based dehazing methods are not fully end-to-end, which still follow the traditional dehazing procedure: first estimate the medium transmission and the atmospheric light, then recover the haze-free image based on the atmospheric scattering model. However, in practice, due to lack of priors and constraints, it is hard to precisely estimate these intermediate parameters. Inaccurate estimation further degrades the performance of de…
Nanjing University of Posts and Telecommunications（南京邮电大学）；ETH Zürich（苏黎世联邦理工学院）；Wuhan University（武汉大学）；Peng Cheng Laboratory（鹏城实验室）；University of Chinese Academy of Sciences（中国科学院大学）；Institute of Automation, CAS（中国科学院自动化研究所）
Synthesizing high-quality realistic images from text descriptions is a challenging task. Existing text-to-image Generative Adversarial Networks generally employ a stacked architecture as the backbone yet still remain three flaws. First, the stacked architecture introduces the entanglements between generators of different image scales. Second, existing studies prefer to apply and fix extra networks in adversarial learning for text-image semantic consistency, which limits the supervision capability of these networks. Third, the cross-modal attention-based text-image fusion that widely adopted by…
In the application of machine learning to remote sensing, labeled data is often scarce or expensive, which impedes the training of powerful models like deep convolutional neural networks. Although unlabeled data is abundant, recent self-supervised learning approaches are ill-suited to the remote sensing domain. In addition, most remote sensing applications currently use only a small subset of the multi-sensor, multi-channel information available, motivating the need for fused multi-sensor representations. We propose a new self-supervised training objective, Contrastive Sensor Fusion, which exp…
GAN-HA: A generative adversarial network with a novel heterogeneous dual-discriminator network and a new attention-based fusion strategy for infrared and visible image fusion
College of New Materials and New Energies, Shenzhen Technology University（深圳技术大学新材料与新能源学院）
Refining raw disparity maps from different algorithms to exploit their complementary advantages is still challenging. Uncertainty estimation and complex disparity relationships among pixels limit the accuracy and robustness of existing methods and there is no standard method for fusion of different kinds of depth data. In this paper, we introduce a new method to fuse disparity maps from different sources, while incorporating supplementary information (intensity, gradient, etc.) into a refiner network to better refine raw disparity inputs. A discriminator network classifies disparities at diffe…
G2HFNet: GeoGran-Aware Hierarchical Feature Fusion Network for Salient Object Detection in Optical Remote Sensing Images
School of Control Science and Engineering, Shandong University（控制科学与工程学院，山东大学）；State Key Laboratory of Autonomous Intelligent Unmanned Systems（自主智能无人系统国家重点实验室）；School of Automation, Hangzhou Dianzi University（自动化学院，杭州电子科技大学）；School of information engineering, Huzhou University（信息工程学院，湖州大学）；School of Data Science, Lingnan University（数据科学学院，岭南大学）
Change Captioning in Remote Sensing: Evolution to SAT-Cap -- A Single-Stage Transformer Approach
Helmholtz-Zentrum Dresden-Rossendorf (HZDR)（德累斯顿-罗森多夫亥姆霍兹中心）；Humboldt-Universität zu Berlin（柏林洪堡大学）；Technical University of Munich（慕尼黑工业大学）；Lancaster University（兰卡斯特大学）
LMFNet: An Efficient Multimodal Fusion Approach for Semantic Segmentation in High-Resolution Remote Sensing
State Key Laboratory of information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University（武汉大学测绘遥感信息工程国家重点实验室）；Electronic Information School, Wuhan University（武汉大学电子信息学院）
