Recent advancements in convolutional neural network (CNN)-based techniques for remote sensing pansharpening have markedly enhanced image quality. However, conventional convolutional modules in these methods have two critical drawbacks. First, the sampling positions in convolution operations are confined to a fixed square window. Second, the number of sampling points is preset and remains unchanged. Given the diverse object sizes in remote sensing images, these rigid parameters lead to suboptimal feature extraction. To overcome these limitations, we introduce an innovative convolutional module, Adaptive Rectangular Convolution (ARConv). ARConv adaptively learns both the height and width of the convolutional kernel and dynamically adjusts the number of sampling points based on the learned scale. This approach enables ARConv to effectively capture scale-specific features of various objects within an image, optimizing kernel sizes and sampling locations. Additionally, we propose ARNet, a network architecture in which ARConv is the primary convolutional module. Extensive evaluations across multiple datasets reveal the superiority of our method in enhancing pansharpening performance over previous techniques. Ablation studies and visualization further confirm the efficacy of ARConv.
National University of Sciences and Technology（国家科技大学）；Technical University of Munich（慕尼黑工业大学）
Pansharpening enhances spatial details of high spectral resolution multispectral images using features of high spatial resolution panchromatic image. There are a number of traditional pansharpening approaches but producing an image exhibiting high spectral and spatial fidelity is still an open problem. Recently, deep learning has been used to produce promising pansharpened images; however, most of these approaches apply similar treatment to both multispectral and panchromatic images by using the same network for feature extraction. In this work, we present present a novel dual attention-based …
Pansharpening is a fundamental issue in remote sensing field. This paper proposes a side information partially guided convolutional sparse coding (SCSC) model for pansharpening. The key idea is to split the low resolution multispectral image into a panchromatic image related feature map and a panchromatic image irrelated feature map, where the former one is regularized by the side information from panchromatic images. With the principle of algorithm unrolling techniques, the proposed model is generalized as a deep neural network, called as SCSC pansharpening neural network (SCSC-PNN). Compared…
Pansharpening is a widely used image enhancement technique for remote sensing. Its principle is to fuse the input high-resolution single-channel panchromatic (PAN) image and low-resolution multi-spectral image and to obtain a high-resolution multi-spectral (HRMS) image. The existing deep learning pansharpening method has two shortcomings. First, features of two input images need to be concatenated along the channel dimension to reconstruct the HRMS image, which makes the importance of PAN images not prominent, and also leads to high computational cost. Second, the implicit information of featu…
University Federico II（费德里科二世大学）；University Parthenope（帕台农佩大学）
We recently proposed a convolutional neural network (CNN) for remote sensing image pansharpening obtaining a significant performance gain over the state of the art. In this paper, we explore a number of architectural and training variations to this baseline, achieving further performance gains with a lightweight network which trains very fast. Leveraging on this latter property, we propose a target-adaptive usage modality which ensures a very good performance also in the presence of a mismatch w.r.t. the training set, and even across different sensors. The proposed method, published online as …
Pansharpening in remote sensing image aims at acquiring a high-resolution multispectral (HRMS) image directly by fusing a low-resolution multispectral (LRMS) image with a panchromatic (PAN) image. The main concern is how to effectively combine the rich spectral information of LRMS image with the abundant spatial information of PAN image. Recently, many methods based on deep learning have been proposed for the pansharpening task. However, these methods usually has two main drawbacks: 1) requiring HRMS for supervised learning; and 2) simply ignoring the latent relation between the MS and PAN ima…
In remote sensing, images acquired by various earth observation satellites tend to have either a high spatial and low spectral resolution or vice versa. Pansharpening is a technique which aims to improve spatial resolution of multispectral image. The challenges involve in the pansharpening are not only to improve the spatial resolution but also to preserve spectral quality of the multispectral image. In this paper, various pansharpening algorithms are discussed and classified based on approaches they have adopted. Using MATLAB image processing toolbox, several state-of-art pan-sharpening algor…
A General Adaptive Dual-level Weighting Mechanism for Remote Sensing Pansharpening
University of Electronic Science and Technology of China（电子科技大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
CMT: Cross Modulation Transformer with Hybrid Loss for Pansharpening
School of Optoelectronic Science and Engineering, University of Electronic Science and Technology of China（电子科技大学光电科学与工程学院）；School of Science, Xihua University（西华大学理学院）；School of Mathematical Sciences, University of Electronic Science and Technology of China（电子科技大学数学科学学院）
Pansharpening is an essential preprocessing step for remote sensing image processing. Although deep learning (DL) approaches performed well on this task, current upsampling methods used in these approaches only utilize the local information of each pixel in the low-resolution multispectral (LRMS) image while neglecting to exploit its global information as well as the cross-modal information of the guiding panchromatic (PAN) image, which limits their performance improvement. To address this issue, this paper develops a novel probability-based global cross-modal upsampling (PGCU) method for pan-…
Recently, convolutional neural networks (CNN) have obtained promising results in single-image SR for hyperspectral pansharpening. However, enhancing CNNs' representation ability with fewer parameters and a shorter prediction time is a challenging and critical task. In this paper, we propose a novel multi-spectral image fusion method using a combination of the previously proposed 3D CNN model VolumeNet and 2.5D texture transfer method using other modality high resolution (HR) images. Since a multi-spectral (MS) image consists of several bands and each band is a 2D image slice, MS images can be …
Dual Modality Prompted Diffusion Priors for Zero Shot Hyperspectral Pansharpening
School of Artificial Intelligence, Beijing Normal University（北京师范大学人工智能学院）；Peking University（北京大学）；National Center for Applied Mathematics Shenzhen (NCAMS), Southern University of Science and Technology（南方科技大学深圳应用数学国家中心）；Department of Automation, Tsinghua University（清华大学自动化系）；National Research Council of Italy, Institute of Integrated Methodologies for Earth Observation (CNR-IMIOT)（意大利国家研究委员会综合地球观测方法研究所）
HetSSNet: Spatial-Spectral Heterogeneous Graph Learning Network for Panchromatic and Multispectral Images Fusion
the School of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；the School of Software Technology, Zhejiang University（浙江大学软件学院）；Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；the Innovation Center of Yangtze River Delta, Zhejiang University（浙江大学长三角创新中心）
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences（中国科学院大学电子电气与通信工程学院）；Unit 31008 of the PLA（中国人民解放军31008部队）
Motivated by the development of deep convolution neural networks (DCNNs), tremendous progress has been gained in the field of aircraft detection. These DCNNs based detectors mainly belong to top-down approaches, which first enumerate massive potential locations of objects with the form of rectangular regions, and then identify whether they are objects or not. Compared with these top-down approaches, this paper shows that aircraft detection via bottom-up approach still performs competitively in the era of deep learning. We present a novel one-stage and anchor-free aircraft detection model in a …
Remote Sensing Technology Institute, German Aerospace Center (DLR)（德国航空航天中心（DLR）遥感技术研究所）；Technical University of Munich（慕尼黑工业大学）
We describe the workflow of a digital surface models (DSMs) refinement algorithm using a hybrid conditional generative adversarial network (cGAN) where the generative part consists of two parallel networks merged at the last stage forming a WNet architecture. The inputs to the so-called WNet-cGAN are stereo DSMs and panchromatic (PAN) half-meter resolution satellite images. Fusing these helps to propagate fine detailed information from a spectral image and complete the missing 3D knowledge from a stereo DSM about building shapes. Besides, it refines the building outlines and edges making them …
The fusion of multispectral and panchromatic images is always dubbed pansharpening. Most of the available deep learning-based pan-sharpening methods sharpen the multispectral images through a one-step scheme, which strongly depends on the reconstruction ability of the network. However, remote sensing images always have large variations, as a result, these one-step methods are vulnerable to the error accumulation and thus incapable of preserving spatial details as well as the spectral information. In this paper, we propose a novel two-step model for pan-sharpening that sharpens the MS image thr…
German Aerospace Center（德国航空航天中心）；Technical University of Munich（慕尼黑工业大学）
Automatic multi-class object detection in remote sensing images in unconstrained scenarios is of high interest for several applications including traffic monitoring and disaster management. The huge variation in object scale, orientation, category, and complex backgrounds, as well as the different camera sensors pose great challenges for current algorithms. In this work, we propose a new method consisting of a novel joint image cascade and feature pyramid network with multi-size convolution kernels to extract multi-scale strong and weak semantic features. These features are fed into rotation-b…
Beihang University（北京航空航天大学）；German Aerospace Center（德国航空航天中心）；Technical University of Munich（慕尼黑工业大学）；University of New Brunswick（新不伦瑞克大学）
Vehicle detection is a significant and challenging task in aerial remote sensing applications. Most existing methods detect vehicles with regular rectangle boxes and fail to offer the orientation of vehicles. However, the orientation information is crucial for several practical applications, such as the trajectory and motion estimation of vehicles. In this paper, we propose a novel deep network, called rotatable region-based residual network (R$^3$-Net), to detect multi-oriented vehicles in aerial images and videos. More specially, R$^3$-Net is utilized to generate rotatable rectangular target…
