Super-resolution of multispectral satellite images using convolutional neural networks
UP42 GmbH（UP42有限公司）
M. U. Müller, N. Ekhtiari, R. M. Almeida, C. Rieke
备注：
To be published in the ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences: https://www.isprs.org/publications/annals.aspx, proceedings of the XXIV ISPRS Congress, 14-20 June 2020, Nice, France
发表信息：
ISPRS Ann. Photogramm. Remote Sens. Spatial Inf. Sci., V-1-2020, 33-40
Super-resolution aims at increasing image resolution by algorithmic means and has progressed over the recent years due to advances in the fields of computer vision and deep learning. Convolutional Neural Networks based on a variety of architectures have been applied to the problem, e.g. autoencoders and residual networks. While most research focuses on the processing of photographs consisting only of RGB color channels, little work can be found concentrating on multi-band, analytic satellite imagery. Satellite images often include a panchromatic band, which has higher spatial resolution but lower spectral resolution than the other bands. In the field of remote sensing, there is a long tradition of applying pan-sharpening to satellite images, i.e. bringing the multispectral bands to the higher spatial resolution by merging them with the panchromatic band. To our knowledge there are so far no approaches to super-resolution which take advantage of the panchromatic band. In this paper we propose a method to train state-of-the-art CNNs using pairs of lower-resolution multispectral and high-resolution pan-sharpened image tiles in order to create super-resolved analytic images. The derived quality metrics show that the method improves information content of the processed images. We compare the results created by four CNN architectures, with RedNet30 performing best.
Taibah University（塔伊巴大学）；Donetsk National Technical University（顿涅茨克国立技术大学）
The paper presents a parallel implementation of existing image fusion methods on a graphical cluster. Parallel implementations of methods based on discrete wavelet transformation (Haars and Daubechies discrete wavelet transform) are developed. Experiments were performed on a cluster using GPU and CPU and performance gains were estimated for the use of the developed parallel implementations to process satellite images from satellite Landsat 7. The implementation on a graphic cluster provides performance improvement from 2 to 18 times. The quality of the considered methods was evaluated by ERGAS…
Various and different methods can be used to produce high-resolution multispectral images from high-resolution panchromatic image (PAN) and low-resolution multispectral images (MS), mostly on the pixel level. However, the jury is still out on the benefits of a fused image compared to its original images. There is also a lack of measures for assessing the objective quality of the spatial resolution for the fusion methods. Therefore, an objective quality of the spatial resolution assessment for fusion images is required. So, this study attempts to develop a new qualitative assessment to evaluate…
Spectral imaging enables spatially-resolved identification of materials in remote sensing, biomedicine, and astronomy. However, acquisition times require balancing spectral and spatial resolution with signal-to-noise. Hyperspectral imaging provides superior material specificity, while multispectral images are faster to collect at greater fidelity. We propose an approach for fusing hyperspectral and multispectral images to provide high-quality hyperspectral output. The proposed optimization leverages the least absolute shrinkage and selection operator (LASSO) to perform variable selection and r…
Most multispectral remote sensors (e.g. QuickBird, IKONOS, and Landsat 7 ETM+) provide low-spatial high-spectral resolution multispectral (MS) or high-spatial low-spectral resolution panchromatic (PAN) images, separately. In order to reconstruct a high-spatial/high-spectral resolution multispectral image volume, either the information in MS and PAN images are fused (i.e. pansharpening) or super-resolution reconstruction (SRR) is used with only MS images captured on different dates. Existing methods do not utilize temporal information of MS and high spatial resolution of PAN images together to …
Hyperspectral and multispectral image fusion with arbitrary resolution through self-supervised representations
Southern University of Science and Technology（南方科技大学）；The Chinese University of Hong Kong（香港中文大学）；University of Electronic Science and Technology of China（电子科技大学）；Hong Kong Baptist University（香港浸会大学）
The University of Adelaide（阿德莱德大学）；University of South Australia（南澳大利亚大学）；York University（约克大学）
Data collected by Earth-observing (EO) satellites are often afflicted by cloud cover. Detecting the presence of clouds -- which is increasingly done using deep learning -- is crucial preprocessing in EO applications. In fact, advanced EO satellites perform deep learning-based cloud detection on board the satellites and downlink only clear-sky data to save precious bandwidth. In this paper, we highlight the vulnerability of deep learning-based cloud detection towards adversarial attacks. By optimising an adversarial pattern and superimposing it into a cloudless scene, we bias the neural network…
Federal University of Santa Catarina（圣卡塔琳娜联邦大学）；Northeastern University（东北大学）；Catholic University of Pelotas（佩洛塔斯天主教大学）
Image fusion combines data from different heterogeneous sources to obtain more precise information about an underlying scene. Hyperspectral-multispectral (HS-MS) image fusion is currently attracting great interest in remote sensing since it allows the generation of high spatial resolution HS images, circumventing the main limitation of this imaging modality. Existing HS-MS fusion algorithms, however, neglect the spectral variability often existing between images acquired at different time instants. This time difference causes variations in spectral signatures of the underlying constituent mate…
Université catholique de Louvain（天主教鲁汶大学）；Université libre de Bruxelles（布鲁塞尔自由大学）；imec（imec研究中心）
This paper introduces two acquisition device architectures for multispectral compressive imaging. Unlike most existing methods, the proposed computational imaging techniques do not include any dispersive element, as they use a dedicated sensor which integrates narrowband Fabry-Pérot spectral filters at the pixel level. The first scheme leverages joint inpainting and super-resolution to fill in those voxels that are missing due to the device's limited pixel count. The second scheme, in link with compressed sensing, introduces spatial random convolutions, but is more complex and may be affected …
Fusing a low-resolution hyperspectral image (HSI) and a high-resolution multispectral image (MSI) of the same scene leads to a super-resolution image (SRI), which is information rich spatially and spectrally. In this paper, we super-resolve the HSI using the graph Laplacian defined on the MSI. Unlike many existing works, we don't assume prior knowledge about the spatial degradation from SRI to HSI, nor a perfectly aligned HSI and MSI pair. Our algorithm progressively alternates between finding the blur kernel and fusing HSI with MSI, generating accurate estimations of the blur kernel and the S…
In this paper a new method of image smoothing for satellite imagery and its applications in environmental remote sensing are presented. This method is based on the global gradient minimization over the whole image. With respect to the image discrete identity, the continuous minimization problem is discretized. Using the finite difference numerical method of differentiation, a simple yet efficient 5*5-pixel template is derived. Convolution of the derived template with the image in different bands results in the discrimination of various image elements. This method is extremely fast, besides bei…
National Institute of Research and Training in Telecommunications (INICTEL-UNI)（国家电信研究与培训学院（INICTEL-UNI））；National University of Engineering（国立工程大学）
Segmenting clouds in high-resolution satellite images is an arduous and challenging task due to the many types of geographies and clouds a satellite can capture. Therefore, it needs to be automated and optimized, specially for those who regularly process great amounts of satellite images, such as governmental institutions. In that sense, the contribution of this work is twofold: We present the CloudPeru2 dataset, consisting of 22,400 images of 512x512 pixels and their respective hand-drawn cloud masks, as well as the proposal of an end-to-end segmentation method for clouds using a Convolutiona…
Does complimentary information from multispectral imaging improve face presentation attack detection?
School of Physical and Applied Sciences, Goa University（果阿大学物理与应用科学学院）；Norwegian University of Science and Technology (NTNU)（挪威科技大学）；Goa Business School, Goa University（果阿大学果阿商学院）；AiBA AS（AiBA AS公司）
The research scope of this paper is to apply spatial object based image analysis (OBIA) method for processing panchromatic multispectral image covering study area of Brussels for urban mapping. The aim is to map different land cover types and more specifically, built-up areas from the very high resolution (VHR) satellite image using OBIA approach. A case study covers urban landscapes in the eastern areas of the city of Brussels, Belgium. Technically, this research was performed in eCognition raster processing software demonstrating excellent results of image segmentation and classification. Th…
Multi-spectral satellite imagery provides valuable data at global scale for many environmental and socio-economic applications. Building supervised machine learning models based on these imagery, however, may require ground reference labels which are not available at global scale. Here, we propose a generative model to produce multi-resolution multi-spectral imagery based on Sentinel-2 data. The resulting synthetic images are indistinguishable from real ones by humans. This technique paves the road for future work to generate labeled synthetic imagery that can be used for data augmentation in …
University of Texas at Arlington（德克萨斯大学阿灵顿分校）；IBM T. J. Watson Research Center（IBM托马斯·J·沃森研究中心）
In this paper, we propose a novel method for image fusion with a high-resolution panchromatic image and a low-resolution multispectral image at the same geographical location. The fusion is formulated as a convex optimization problem which minimizes a linear combination of a least-squares fitting term and a dynamic gradient sparsity regularizer. The former is to preserve accurate spectral information of the multispectral image, while the latter is to keep sharp edges of the high-resolution panchromatic image. We further propose to simultaneously register the two images during the fusing proces…
Shiraz University of Technology（设拉子理工大学）；Persian Gulf University（波斯湾大学）；Yazd Branch, Islamic Azad University（伊斯兰阿扎德大学亚兹德分校）；Najafabad Branch, Islamic Azad University（伊斯兰阿扎德大学纳杰法巴德分校）
In interpretation of remote sensing images, it is possible that some images which are supplied by different sensors become incomprehensible. For better visual perception of these images, it is essential to operate series of pre-processing and elementary corrections and then operate a series of main processing steps for more precise analysis on the images. There are several approaches for processing which are depended on the type of remote sensing images. The discussed approach in this article, i.e. image fusion, is the use of natural colors of an optical image for adding color to a grayscale s…
Scientific Research Group in Egypt (SRGE)（埃及科学研究组）；Benha University（本哈大学）；Damietta University（达米埃塔大学）；Cairo University（开罗大学）
Remote sensing image fusion is an effective way to use a large volume of data from multisensor images. Most earth satellites such as SPOT, Landsat 7, IKONOS and QuickBird provide both panchromatic (Pan) images at a higher spatial resolution and multispectral (MS) images at a lower spatial resolution and many remote sensing applications require both high spatial and high spectral resolutions, especially for GIS based applications. An effective image fusion technique can produce such remotely sensed images. Image fusion is the combination of two or more different images to form a new image by us…
University of Sannio（桑尼奥大学）；Italian Space Agency（意大利空间局）
In recent years, the growth of Machine Learning (ML) algorithms has raised the number of studies including their applicability in a variety of different scenarios. Among all, one of the hardest ones is the aerospace, due to its peculiar physical requirements. In this context, a feasibility study and a first prototype for an Artificial Intelligence (AI) model to be deployed on board satellites are presented in this work. As a case study, the detection of volcanic eruptions has been investigated as a method to swiftly produce alerts and allow immediate interventions. Two Convolutional Neural Net…
Xi’an Jiaotong University（西安交通大学）；Harbin Institute of Technology（哈尔滨工业大学）
Hyperspectral imaging can help better understand the characteristics of different materials, compared with traditional image systems. However, only high-resolution multispectral (HrMS) and low-resolution hyperspectral (LrHS) images can generally be captured at video rate in practice. In this paper, we propose a model-based deep learning approach for merging an HrMS and LrHS images to generate a high-resolution hyperspectral (HrHS) image. In specific, we construct a novel MS/HS fusion model which takes the observation models of low-resolution images and the low-rankness knowledge along the spec…
RoughNet: Mapping Arctic Sea Ice Roughness Using Diffusion-Based Super-Resolution of Satellite Imagery
University College London（伦敦大学学院）；Alfred Wegener Institute Helmholtz Centre for Polar and Marine Research（阿尔弗雷德·韦格纳极地与海洋研究所亥姆霍兹中心）；University of Victoria（维多利亚大学）
The Ohio State University（俄亥俄州立大学）；Byrd Polar and Climate Research Center（伯德极地与气候研究中心）
Remote sensing of the Earth's surface water is critical in a wide range of environmental studies, from evaluating the societal impacts of seasonal droughts and floods to the large-scale implications of climate change. Consequently, a large literature exists on the classification of water from satellite imagery. Yet, previous methods have been limited by 1) the spatial resolution of public satellite imagery, 2) classification schemes that operate at the pixel level, and 3) the need for multiple spectral bands. We advance the state-of-the-art by 1) using commercial imagery with panchromatic and …
University of South Carolina（南卡罗来纳大学）；East China Normal University（华东师范大学）
Nighttime satellite imagery has been applied in a wide range of fields. However, our limited understanding of how observed light intensity is formed and whether it can be simulated greatly hinders its further application. This study explores the potential of conditional Generative Adversarial Networks (cGAN) in translating multispectral imagery to nighttime imagery. A popular cGAN framework, pix2pix, was adopted and modified to facilitate this translation using gridded training image pairs derived from Landsat 8 and Visible Infrared Imaging Radiometer Suite (VIIRS). The results of this study p…
Satellite imaging has a central role in monitoring, detecting and estimating the intensity of key natural phenomena. One important feature of satellite images is the trade-off between spatial/spectral resolution and their revisiting time, a consequence of design and physical constraints imposed by satellite orbit among other technical limitations. In this paper, we focus on fusing multi-temporal, multi-spectral images where data acquired from different instruments with different spatial resolutions is used. We leverage the spatial relationship between images at multiple modalities to generate …
As of any other satellite images, LAPAN-A3/IPB multispectral images suffered from both geometric and radiometric distortions which need to be corrected. LAPAN as satellite owner has developed image preprocessing algorithm to process raw image into systematically corrected image. This research aims to evaluate the performance of the developed algorithm, particularly the performance of lens vignetting and band co-registration correction as well as the performance of direct image georeferencing. Lens vignetting distortion occurs on image was corrected by using pre-flight calibration data, while c…
Spectral Super-Resolution via Adversarial Unfolding and Data-Driven Spectrum Regularization: From Multispectral Satellite Data to NASA Hyperspectral Image
NASA Ames Research Center（美国国家航空航天局艾姆斯研究中心）；Bay Area Environmental Research Institute（湾区环境研究所）；Microsoft Research AI（微软研究院人工智能）；California State University Monterey Bay（加利福尼亚州立大学蒙特雷湾分校）
Earth observing satellites carrying multi-spectral sensors are widely used to monitor the physical and biological states of the atmosphere, land, and oceans. These satellites have different vantage points above the earth and different spectral imaging bands resulting in inconsistent imagery from one to another. This presents challenges in building downstream applications. What if we could generate synthetic bands for existing satellites from the union of all domains? We tackle the problem of generating synthetic spectral imagery for multispectral sensors as an unsupervised image-to-image trans…
Indraprastha Institute of Information Technology, Delhi（德里英迪拉普拉斯塔信息技术学院）；M S Swaminathan Research Foundation（M·S·斯瓦米纳坦研究基金会）
The integration of the modern Machine Learning (ML) models into remote sensing and agriculture has expanded the scope of the application of satellite images in the agriculture domain. In this paper, we present how the accuracy of crop type identification improves as we move from medium-spatiotemporal-resolution (MSTR) to high-spatiotemporal-resolution (HSTR) satellite images. We further demonstrate that high spectral resolution in satellite imagery can improve prediction performance for low spatial and temporal resolutions (LSTR) images. The F1-score is increased by 7% when using multispectral…
This paper presents a neural-network-based solution to recover pixels occluded by clouds in satellite images. We leverage radio frequency (RF) signals in the ultra/super-high frequency band that penetrate clouds to help reconstruct the occluded regions in multispectral images. We introduce the first multi-modal multi-temporal cloud removal model. Our model uses publicly available satellite observations and produces daily cloud-free images. Experimental results show that our system significantly outperforms baselines by 8dB in PSNR. We also demonstrate use cases of our system in digital agricul…
We present a procedure for assessing the urban exposure and seismic vulnerability that integrates LiDAR data with aerial and satellite images. It comprises three phases: first, we segment the satellite image to divide the study area into different urban patterns. Second, we extract building footprints and attributes that represent the type of building of each urban pattern. Finally, we assign the seismic vulnerability to each building using different machine-learning techniques: Decision trees, SVM, logistic regression and Bayesian networks. We apply the procedure to 826 buildings in the city …
Unmanned aerial vehicles (UAV) are used in precision agriculture (PA) to enable aerial monitoring of farmlands. Intelligent methods are required to pinpoint weed infestations and make optimal choice of pesticide. UAV can fly a multispectral camera and collect data. However, the classification of multispectral images using supervised machine learning algorithms such as convolutional neural networks (CNN) requires large amount of training data. This is a common drawback in deep learning we try to circumvent making use of a semi-supervised generative adversarial networks (GAN), providing a pixel-…
Real-time satellite imaging has a central role in monitoring, detecting and estimating the intensity of key natural phenomena such as floods, earthquakes, etc. One important constraint of satellite imaging is the trade-off between spatial/spectral resolution and their revisiting time, a consequence of design and physical constraints imposed by satellite orbit among other technical limitations. In this paper, we focus on fusing multi-temporal, multi-spectral images where data acquired from different instruments with different spatial resolutions is used. We leverage the spatial relationship bet…
Universidade de Pernambuco（伯南布哥大学）；Universidade Federal de Campina Grande（坎皮纳格兰德联邦大学）；Universidade Federal de Pernambuco（伯南布哥联邦大学）；North Carolina State University（北卡罗来纳州立大学）
Multispectral image analysis is a relatively promising field of research with applications in several areas, such as medical imaging and satellite monitoring. A considerable number of current methods of analysis are based on parametric statistics. Alternatively, some methods in Computational Intelligence are inspired by biology and other sciences. Here we claim that Philosophy can be also considered as a source of inspiration. This work proposes the Objective Dialectical Method (ODM): a method for classification based on the Philosophy of Praxis. ODM is instrumental in assembling evolvable mat…
Instituto de Ingeniería（工程研究所）；Instituto Nacional de Bioingeniería, Universidad Central de Venezuela（委内瑞拉中央大学国家生物工程研究所）；Algorithmic Nature Lab, LABORES for the Natural and Digital Sciences（LABORES自然与数字科学 Algorithmic Nature 实验室）
The segmentation of satellite images is a necessary step to perform object-oriented image classification, which has become relevant due to its applicability on images with a high spatial resolution. To perform object-oriented image classification, the studied image must first be segmented in uniform regions. This segmentation requires manual work by an expert user, who must exhaustively explore the image to establish thresholds that generate useful and representative segments without oversegmenting and without discarding representative segments. We propose a technique that automatically segmen…
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；Univ. Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；CNRS（法国国家科学研究中心）；Grenoble INP（格勒诺布尔理工学院）；University of Chinese Academy of Sciences（中国科学院大学）；University of Pavia（帕维亚大学）；INRIA（法国国家信息与自动化研究所）
Extensive attention has been widely paid to enhance the spatial resolution of hyperspectral (HS) images with the aid of multispectral (MS) images in remote sensing. However, the ability in the fusion of HS and MS images remains to be improved, particularly in large-scale scenes, due to the limited acquisition of HS images. Alternatively, we super-resolve MS images in the spectral domain by the means of partially overlapped HS images, yielding a novel and promising topic: spectral superresolution (SSR) of MS imagery. This is challenging and less investigated task due to its high ill-posedness i…
Pan-sharpening is an important technique for remote sensing imaging systems to obtain high resolution multispectral images. Recently, deep learning has become the most popular tool for pan-sharpening. This paper develops a model-based deep pan-sharpening approach. Specifically, two optimization problems regularized by the deep prior are formulated, and they are separately responsible for the generative models for panchromatic images and low resolution multispectral images. Then, the two problems are solved by a gradient projection algorithm, and the iterative steps are generalized into two net…
This paper presents a generative model method for multispectral image fusion in remote sensing which is trained without supervision. This method eases the supervision of learning and it also considers a multi-objective loss function to achieve image fusion. The loss function incorporates both spectral and spatial distortions. Two discriminators are designed to minimize the spectral and spatial distortions of the generative output. Extensive experimentations are conducted using three public domain datasets. The comparison results across four reduced-resolution and three full-resolution objectiv…
Pansharpening aims to fuse panchromatic and multispectral images from the satellite to generate images with both high spatial and spectral resolution. With the successful applications of deep learning in the computer vision field, a lot of scholars have proposed many convolutional neural networks (CNNs) to solve the pansharpening task. These pansharpening networks focused on various distinctive structures of CNNs, and most of them are trained by L2 loss between fused images and simulated desired multispectral images. However, L2 loss is designed to directly minimize the difference of spectral …
Skolkovo Institute of Science and Technology（斯科尔科沃科学技术学院）
The near-infrared (NIR) spectral range (from 780 to 2500 nm) of the multispectral remote sensing imagery provides vital information for the landcover classification, especially concerning the vegetation assessment. Despite the usefulness of NIR, common RGB is not always accompanied by it. Modern achievements in image processing via deep neural networks allow generating artificial spectral information, such as for the image colorization problem. In this research, we aim to investigate whether this approach can produce not only visually similar images but also an artificial spectral band that ca…
