In the field of fusing multi-spectral and panchromatic images (Pan-sharpening), the impressive effectiveness of deep neural networks has been recently employed to overcome the drawbacks of traditional linear models and boost the fusing accuracy. However, to the best of our knowledge, existing research works are mainly based on simple and flat networks with relatively shallow architecture, which severely limited their performances. In this paper, the concept of residual learning has been introduced to form a very deep convolutional neural network to make a full use of the high non-linearity of deep learning models. By both quantitative and visual assessments on a large number of high quality multi-spectral images from various sources, it has been supported that our proposed model is superior to all mainstream algorithms included in the comparison, and achieved the highest spatial-spectral unified accuracy.
Pan-sharpening is a fundamental and significant task in the field of remote sensing imagery processing, in which high-resolution spatial details from panchromatic images are employed to enhance the spatial resolution of multi-spectral (MS) images. As the transformation from low spatial resolution MS image to high-resolution MS image is complex and highly non-linear, inspired by the powerful representation for non-linear relationships of deep neural networks, we introduce multi-scale feature extraction and residual learning into the basic convolutional neural network (CNN) architecture and prop…
This paper addresses the problem of remote sensing image pan-sharpening from the perspective of generative adversarial learning. We propose a novel deep neural network based method named PSGAN. To the best of our knowledge, this is one of the first attempts at producing high-quality pan-sharpened images with GANs. The PSGAN consists of two components: a generative network (i.e., generator) and a discriminative network (i.e., discriminator). The generator is designed to accept panchromatic (PAN) and multispectral (MS) images as inputs and maps them to the desired high-resolution (HR) MS images …
University of Naples Federico II（那不勒斯费德里科二世大学）；University of Modena and Reggio Emilia（摩德纳和雷焦艾米利亚大学）
Multispectral (MS) image panchromatic (PAN) sharpening algorithms proposed to the remote sensing community are ever increasing in number and variety. Their aim is to sharpen a coarse spatial resolution MS image with a fine spatial resolution PAN image acquired simultaneously by a spaceborne or airborne Earth observation (EO) optical imaging sensor pair. Unfortunately, to date, no standard evaluation procedure for MS image PAN sharpening outcome and process is community agreed upon, in contrast with the Quality Assurance Framework for Earth Observation (QA4EO) guidelines proposed by the intergo…
Magellium；ITAV-USR3505, université de Toulouse（图卢兹大学ITAV-USR3505）；CNES（法国国家空间研究中心）
In this paper, we address the issue of hyperspectral pan-sharpening, which consists in fusing a (low spatial resolution) hyperspectral image HX and a (high spatial resolution) panchromatic image P to obtain a high spatial resolution hyperspectral image. The problem is addressed under a variational convex constrained formulation. The objective favors high resolution spectral bands with level lines parallel to those of the panchromatic image. This term is balanced with a total variation term as regularizer. Fit-to-P data and fit-to-HX data constraints are effectively considered as mathematical c…
Frequency-Adaptive Pan-Sharpening with Mixture of Experts
Hefei Institutes of Physical Science, Chinese Academy of Sciences（中国科学院合肥物质科学研究院）；University of Science and Technology of China（中国科学技术大学）；Nanyang Technological University（南洋理工大学）
Pan-sharpening is an important technique for remote sensing imaging systems to obtain high resolution multispectral images. Recently, deep learning has become the most popular tool for pan-sharpening. This paper develops a model-based deep pan-sharpening approach. Specifically, two optimization problems regularized by the deep prior are formulated, and they are separately responsible for the generative models for panchromatic images and low resolution multispectral images. Then, the two problems are solved by a gradient projection algorithm, and the iterative steps are generalized into two net…
The State Key Laboratory of Virtual Reality Technology and Systems, Beihang University（北京航空航天大学虚拟现实技术与系统国家重点实验室）；Hangzhou Innovation Institute, Beihang University（北京航空航天大学杭州创新研究院）
Pan-sharpening aims at producing a high-resolution (HR) multi-spectral (MS) image from a low-resolution (LR) multi-spectral (MS) image and its corresponding panchromatic (PAN) image acquired by a same satellite. Inspired by a new fashion in recent deep learning community, we propose a novel Transformer based model for pan-sharpening. We explore the potential of Transformer in image feature extraction and fusion. Following the successful development of vision transformers, we design a two-stream network with the self-attention to extract the modality-specific features from the PAN and MS modali…
CC-Pan: Channel-wise Compression based Diffusion for Efficient Pan-Sharpening
School of Cyberspace Security, Northwestern Polytechnical University（网络安全学院，西北工业大学）；Nanjing University of Science and Technology（南京理工大学）；School of Computer Science, Nanjing University of Information Science and Technology（计算机科学学院，南京信息工程大学）
Pan-LUT: Efficient Pan-sharpening via Learnable Look-Up Tables
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University（教育部多媒体可信感知与高效计算重点实验室，厦门大学）；ByteDance（字节跳动）；The Chinese University of Hong Kong（香港中文大学）；Duke University（杜克大学）；University of Washington（华盛顿大学）
Remote sensing image fusion (also known as pan-sharpening) aims at generating high resolution multi-spectral (MS) image from inputs of a high spatial resolution single band panchromatic (PAN) image and a low spatial resolution multi-spectral image. Inspired by the astounding achievements of convolutional neural networks (CNNs) in a variety of computer vision tasks, in this paper, we propose a two-stream fusion network (TFNet) to address the problem of pan-sharpening. Unlike previous CNN based methods that consider pan-sharpening as a super resolution problem and perform pan-sharpening in pixel…
An ideal fusion method preserves the Spectral information in fused image and adds spatial information to it with no spectral distortion. Recently wavelet kalman filter method is proposed which uses ARSIS concept to fuses MS and PAN images. This method is applied in a multiscale version, i.e. the variable index is scale instead of time. With the aim of fusion we present a more detailed study on this model and discuss about rationality of its assumptions such as first order markov model and Gaussian distribution of the posterior density. Finally, we propose a method using wavelet Kalman Particle…
Pan-sharpening is a process of merging a high-resolution (HR) panchromatic (PAN) image and its corresponding low-resolution (LR) multi-spectral (MS) image to create an HR-MS and pan-sharpened image. However, due to the different sensors' locations, characteristics and acquisition time, PAN and MS image pairs often tend to have various amounts of misalignment. Conventional deep-learning-based methods that were trained with such misaligned PAN-MS image pairs suffer from diverse artifacts such as double-edge and blur artifacts in the resultant PAN-sharpened images. In this paper, we propose a nov…
Rethinking Pan-sharpening: A New Training Process for Full-Resolution Generalization
Hefei University of Technology（合肥工业大学）；University of Science and Technology of China（中国科学技术大学）；Hefei Institutes of Physical Science, Chinese Academy of Sciences（中国科学院合肥物质科学研究院）
Korea Advanced Institute of Science and Technology（韩国科学技术院）；Korea Aerospace Research Institute（韩国航空宇宙研究院）
Recently, many deep-learning-based pan-sharpening methods have been proposed for generating high-quality pan-sharpened (PS) satellite images. These methods focused on various types of convolutional neural network (CNN) structures, which were trained by simply minimizing a spectral loss between network outputs and the corresponding high-resolution multi-spectral (MS) target images. However, due to different sensor characteristics and acquisition times, high-resolution panchromatic (PAN) and low-resolution MS image pairs tend to have large pixel misalignments, especially for moving objects in th…
Wuhan University（武汉大学）；University of Arkansas（阿肯色大学）；Wuhan Institute of Technology（武汉工程大学）
Most existing deep learning-based pan-sharpening methods have several widely recognized issues, such as spectral distortion and insufficient spatial texture enhancement, we propose a novel pan-sharpening convolutional neural network based on a high-pass modification block. Different from existing methods, the proposed block is designed to learn the high-pass information, leading to enhance spatial information in each band of the multi-spectral-resolution images. To facilitate the generation of visually appealing pan-sharpened images, we propose a perceptual loss function and further optimize t…
Deep learning based pan-sharpening has received significant research interest in recent years. Most of existing methods fall into the supervised learning framework in which they down-sample the multi-spectral (MS) and panchromatic (PAN) images and regard the original MS images as ground truths to form training samples. Although impressive performance could be achieved, they have difficulties generalizing to the original full-scale images due to the scale gap, which makes them lack of practicability. In this paper, we propose an unsupervised generative adversarial framework that learns from the…
The fusion of multispectral and panchromatic images is always dubbed pansharpening. Most of the available deep learning-based pan-sharpening methods sharpen the multispectral images through a one-step scheme, which strongly depends on the reconstruction ability of the network. However, remote sensing images always have large variations, as a result, these one-step methods are vulnerable to the error accumulation and thus incapable of preserving spatial details as well as the spectral information. In this paper, we propose a novel two-step model for pan-sharpening that sharpens the MS image thr…
Multigrain-aware Semantic Prototype Scanning and Tri-Token Prompt Learning Embraced High-Order RWKV for Pan-Sharpening
Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）；Yunnan Normal University, Ministry of Education（云南师范大学教育部）；Xidian University（西安电子科技大学）；University of Science and Technology of China（中国科学技术大学）；The Hong Kong University of Science and Technology（香港理工大学）；MoE Key Laboratory of Information Technology（教育部信息科学技术重点实验室）
Beihang University（北京航空航天大学）；Hangzhou Innovation Institute, Beihang University（北京航空航天大学杭州创新研究院）
Pan-sharpening aims at fusing a low-resolution (LR) multi-spectral (MS) image and a high-resolution (HR) panchromatic (PAN) image acquired by a satellite to generate an HR MS image. Many deep learning based methods have been developed in the past few years. However, since there are no intended HR MS images as references for learning, almost all of the existing methods down-sample the MS and PAN images and regard the original MS images as targets to form a supervised setting for training. These methods may perform well on the down-scaled images, however, they generalize poorly to the full-resol…
Southeast University（东南大学）；University of Electronic Science and Technology of China（电子科技大学）；University of Science and Technology of China（中国科学技术大学）；Hong Kong Polytechnic University（香港理工大学）；Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）
Dominant pan-sharpening frameworks simply concatenate the MS stream and the PAN stream once at a specific level. This way of fusion neglects the multi-level spectral-spatial correlation between the two streams, which is vital to improving the fusion performance. In consideration of this, we propose a Multi-level and Enhanced Spectral-Spatial Fusion Network (MESSFN) with the following innovations: First, to fully exploit and strengthen the above correlation, a Hierarchical Multi-level Fusion Architecture (HMFA) is carefully designed. A novel Spectral-Spatial (SS) stream is established to hierar…
Yingcai Honors College（英才荣誉学院）；University of Electronic Science and Technology of China（电子科技大学）；School of Mathematical Sciences（数学科学学院）；Univ. Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；Inria（法国国家信息与自动化研究所）；CNRS（法国国家科学研究中心）；Grenoble INP（格勒诺布尔理工学院）；LJK（让·昆兹实验室）；National Research Council - Institute of Methodologies for Environmental Analysis（国家研究委员会 - 环境分析方法研究所）；CNR-IMAA（国家研究委员会 - 环境分析方法研究所）
Pansharpening refers to the fusion of a panchromatic image with a high spatial resolution and a multispectral image with a low spatial resolution, aiming to obtain a high spatial resolution multispectral image. In this paper, we propose a novel deep neural network architecture with level-domain based loss function for pansharpening by taking into account the following double-type structures, \emph{i.e.,} double-level, double-branch, and double-direction, called as triple-double network (TDNet). By using the structure of TDNet, the spatial details of the panchromatic image can be fully exploite…
Hefei Institute of Physical Science, Chinese Academy of Sciences（中国科学院合肥物质科学研究院）；University of Science and Technology of China（中国科学技术大学）；Nanyang Technological University（南洋理工大学）；Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）
Panchromatic (PAN) and multi-spectral (MS) image fusion, named Pan-sharpening, refers to super-resolve the low-resolution (LR) multi-spectral (MS) images in the spatial domain to generate the expected high-resolution (HR) MS images, conditioning on the corresponding high-resolution PAN images. In this paper, we present a simple yet effective \textit{alternating reverse filtering network} for pan-sharpening. Inspired by the classical reverse filtering that reverses images to the status before filtering, we formulate pan-sharpening as an alternately iterative reverse filtering process, which fus…
Remote sensing image fusion technology (pan-sharpening) is an important means to improve the information capacity of remote sensing images. Inspired by the efficient arameter space posteriori sampling of Bayesian neural networks, in this paper we propose a Bayesian Generative Adversarial Network based on Preconditioned Stochastic Gradient Langevin Dynamics (PGSLD-BGAN) to improve pan-sharpening tasks. Unlike many traditional generative models that consider only one optimal solution (might be locally optimal), the proposed PGSLD-BGAN performs Bayesian inference on the network parameters, and ex…
In remote sensing, images acquired by various earth observation satellites tend to have either a high spatial and low spectral resolution or vice versa. Pansharpening is a technique which aims to improve spatial resolution of multispectral image. The challenges involve in the pansharpening are not only to improve the spatial resolution but also to preserve spectral quality of the multispectral image. In this paper, various pansharpening algorithms are discussed and classified based on approaches they have adopted. Using MATLAB image processing toolbox, several state-of-art pan-sharpening algor…
Pansharpening is an essential preprocessing step for remote sensing image processing. Although deep learning (DL) approaches performed well on this task, current upsampling methods used in these approaches only utilize the local information of each pixel in the low-resolution multispectral (LRMS) image while neglecting to exploit its global information as well as the cross-modal information of the guiding panchromatic (PAN) image, which limits their performance improvement. To address this issue, this paper develops a novel probability-based global cross-modal upsampling (PGCU) method for pan-…
A Deep Single Image Rectification Approach for Pan-Tilt-Zoom Cameras
School of Computer Science, Hubei University of Technology（湖北工业大学计算机学院）；Hubei Key Laboratory of Green Intelligent Computing Power Network（湖北绿色智能算力网络重点实验室）；School of Surveying and Mapping, Wuhan University（武汉大学测绘学院）；Wuhan Tianjihang Information Technology Co., Ltd.（武汉天起航信息技术有限公司）
An ideal fusion method preserves the Spectral information in fused image and adds spatial information to it with no spectral distortion. Among the existing fusion algorithms, the contourlet-based fusion method is the most frequently discussed one in recent publications, because the contourlet has the ability to capture and link the point of discontinuities to form a linear structure. The Brovey is a popular pan-sharpening method owing to its efficiency and high spatial resolution. This method can be explained by mathematical model of optical remote sensing sensors. This study presents a new fu…
Super-resolution aims at increasing image resolution by algorithmic means and has progressed over the recent years due to advances in the fields of computer vision and deep learning. Convolutional Neural Networks based on a variety of architectures have been applied to the problem, e.g. autoencoders and residual networks. While most research focuses on the processing of photographs consisting only of RGB color channels, little work can be found concentrating on multi-band, analytic satellite imagery. Satellite images often include a panchromatic band, which has higher spatial resolution but lo…
Nanyang Technological University（南洋理工大学）；National University of Singapore（新加坡国立大学）；Risk Management Institute, National University of Singapore（新加坡国立大学风险管理研究所）；National Taiwan University（台湾大学）；National University Hospital（新加坡国立大学医院）；National University Cancer Institute（新加坡国立大学癌症研究所）
We present a novel spectral machine learning (SML) method in screening for pancreatic mass using CT imaging. Our algorithm is trained with approximately 30,000 images from 250 patients (50 patients with normal pancreas and 200 patients with abnormal pancreas findings) based on public data sources. A test accuracy of 94.6 percents was achieved in the out-of-sample diagnosis classification based on a total of approximately 15,000 images from 113 patients, whereby 26 out of 32 patients with normal pancreas and all 81 patients with abnormal pancreas findings were correctly diagnosed. SML is able t…
During the coronavirus disease 2019 (COVID-19) pandemic, rapid and accurate triage of patients at the emergency department is critical to inform decision-making. We propose a data-driven approach for automatic prediction of deterioration risk using a deep neural network that learns from chest X-ray images and a gradient boosting model that learns from routine clinical variables. Our AI prognosis system, trained using data from 3,661 patients, achieves an area under the receiver operating characteristic curve (AUC) of 0.786 (95% CI: 0.745-0.830) when predicting deterioration within 96 hours. Th…
Amirkabir University of Technology（阿米尔卡比尔理工大学）；Semnan University（塞姆南大学）
In late 2019 and after COVID-19 pandemic in the world, many researchers and scholars have tried to provide methods for detection of COVID-19 cases. Accordingly, this study focused on identifying COVID-19 cases from chest X-ray images. In this paper, a novel approach to diagnosing coronavirus disease from X-ray images was proposed. In the proposed method, DenseNet169 deep neural network was used to extract the features of X-ray images taken from the patients' chest and the extracted features were then given as input to the Extreme Gradient Boosting (XGBoost) algorithm so that it could perform t…
Hefei Institute of Physical Science, Chinese Academy of Sciences（中国科学院合肥物质科学研究院）；University of Science and Technology of China（中国科学技术大学）；Nanjing University of Science and Technology（南京理工大学）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；Xi’an Jiaotong University（西安交通大学）
Guided image super-resolution (GISR) aims to obtain a high-resolution (HR) target image by enhancing the spatial resolution of a low-resolution (LR) target image under the guidance of a HR image. However, previous model-based methods mainly takes the entire image as a whole, and assume the prior distribution between the HR target image and the HR guidance image, simply ignoring many non-local common characteristics between them. To alleviate this issue, we firstly propose a maximal a posterior (MAP) estimation model for GISR with two types of prior on the HR target image, i.e., local implicit …
Corona virus or COVID-19 is a pandemic illness, which has influenced more than million of causalities worldwide and infected a few large number of individuals .Innovative instrument empowering quick screening of the COVID-19 contamination with high precision can be critically useful to the medical care experts. The primary clinical device presently being used for the analysis of COVID-19 is the Reverse record polymerase chain response as known as RT-PCR, which is costly, less-delicate and requires specific clinical work force. X-Ray imaging is an effectively available apparatus that can be a g…
Classification of COVID-19 in Chest X-ray Images Using Fusion of Deep Features and LightGBM
Amirkabir University of Technology（阿米尔卡比尔理工大学）；Semnan University（塞姆南大学）；University of New Mexico（新墨西哥大学）；Technical and Vocational University (TVU)（技术与职业大学（TVU））
