Thanks to their temporal-spatial coverage and free access, Sentinel-2 images are very interesting for the community. However, a relatively coarse spatial resolution, compared to that of state-of-the-art commercial products, motivates the study of super-resolution techniques to mitigate such a limitation. Specifically, thirtheen bands are sensed simultaneously but at different spatial resolutions: 10, 20, and 60 meters depending on the spectral location. Here, building upon our previous convolutional neural network (CNN) based method, we propose an improved CNN solution to super-resolve the 20-m resolution bands benefiting spatial details conveyed by the accompanying 10-m spectral bands.
Remote Sensing applications can benefit from a relatively fine spatial resolution multispectral (MS) images and a high revisit frequency ensured by the twin satellites Sentinel-2. Unfortunately, only four out of thirteen bands are provided at the highest resolution of 10 meters, and the others at 20 or 60 meters. For instance the Short-Wave Infrared (SWIR) bands, provided at 20 meters, are very useful to detect active fires. Aiming to a more detailed Active Fire Detection (AFD) maps, we propose a super-resolution data fusion method based on Convolutional Neural Network (CNN) to move towards th…
Self-Supervised Super-Resolution for Sentinel-5P Hyperspectral Images
Université d’Orléans, Université de Tours, CNRS, IDP, UMR 7013（奥尔良大学、图尔大学、法国国家科学研究中心、IDP、UMR 7013）；Faculty of Mathematical Sciences, University of Khartoum（数学科学学院，喀土穆大学）；ENS Lyon, France（里昂高等师范学院，法国）；Laboratory of Physics and Chemistry of the Environment and Space (LPC2E), CNRS UMR 7328, University of Orléans, France（环境与空间物理化学实验室（LPC2E），法国国家科学研究中心UMR 7328，奥尔良大学，法国）；Institut universitaire de France (IUF), France（法国国家研究院（IUF），法国）
ETH Zurich（苏黎世联邦理工学院）；Instituto de Telecomunicações（电信研究所）；Instituto Superior Técnico（高等技术学院）；Universidade de Lisboa（里斯本大学）
The Sentinel-2 satellite mission delivers multi-spectral imagery with 13 spectral bands, acquired at three different spatial resolutions. The aim of this research is to super-resolve the lower-resolution (20 m and 60 m Ground Sampling Distance - GSD) bands to 10 m GSD, so as to obtain a complete data cube at the maximal sensor resolution. We employ a state-of-the-art convolutional neural network (CNN) to perform end-to-end upsampling, which is trained with data at lower resolution, i.e., from 40->20 m, respectively 360->60 m GSD. In this way, one has access to a virtually infinite amount of tr…
On The Role of Alias and Band-Shift for Sentinel-2 Super-Resolution
Université Paris-Saclay（巴黎萨克雷大学）；CNRS（法国国家科学研究中心）；ENS Paris-Saclay（巴黎萨克雷高等师范学院）；Kayrros SAS（Kayrros公司）；Univ Gustave Eiffel（古斯塔夫·埃菲尔大学）；Institut Denis Poisson（德尼·泊松研究所）；Université de Tours（图尔大学）；Institut universitaire de France (IUF)（法国大学研究院）
University of Oxford（牛津大学）；University of Valencia（瓦伦西亚大学）
High resolution remote sensing imagery is used in broad range of tasks, including detection and classification of objects. High-resolution imagery is however expensive, while lower resolution imagery is often freely available and can be used by the public for range of social good applications. To that end, we curate a multi-spectral multi-image super-resolution dataset, using PlanetScope imagery from the SpaceNet 7 challenge as the high resolution reference and multiple Sentinel-2 revisits of the same imagery as the low-resolution imagery. We present the first results of applying multi-image s…
Beyond Pretty Pictures: Combined Single- and Multi-Image Super-resolution for Sentinel-2 Images
Asian Development Bank, Philippines（亚洲开发银行，菲律宾）；GeoVille Information Systems and Data Processing GmbH, A-6020 Innsbruck, Austria（GeoVille信息系统和数据处理公司，奥地利）；ETH Zürich, Switzerland（苏黎世联邦理工学院，瑞士）
Cross-sensor super-resolution of irregularly sampled Sentinel-2 time series
Université Bretagne Sud（南布列塔尼大学）；IRISA（IRISA研究所）；Université Gustave Eiffel（Gustave Eiffel大学）；ENSG（法国国家地理科学学校）；IGN（法国国家地理研究院）；LASTIG（LASTIG实验室）；Conservatoire national des arts et métiers（法国国立工艺学院）；CEDRIC（CEDRIC研究所）；Image Processing Laboratory (IPL)（图像处理实验室）；Universitat de València（瓦伦西亚大学）
Technical University of Munich（慕尼黑工业大学）；Information Engineering University（信息工程大学）；Ruhr-University Bochum（波鸿鲁尔大学）；German Aerospace Center（德国航空航天中心）
As a unique classification scheme for urban forms and functions, the local climate zone (LCZ) system provides essential general information for any studies related to urban environments, especially on a large scale. Remote sensing data-based classification approaches are the key to large-scale mapping and monitoring of LCZs. The potential of deep learning-based approaches is not yet fully explored, even though advanced convolutional neural networks (CNNs) continue to push the frontiers for various computer vision tasks. One reason is that published studies are based on different datasets, usua…
The increasing spatial and temporal resolution of globally available satellite images, such as provided by Sentinel-2, creates new possibilities for researchers to use freely available multi-spectral optical images, with decametric spatial resolution and more frequent revisits for remote sensing applications such as land cover and crop classification (LC&CC), agricultural monitoring and management, environment monitoring. Existing solutions dedicated to cropland mapping can be categorized based on per-pixel based and object-based. However, it is still challenging when more classes of agricultu…
Institute of Informatics and Telecommunications, NCSR Demokritos（希腊德谟克利特国家研究中心信息与通信研究所）；University of West Attica（西阿提卡大学）；Helenic Air Force Academy（希腊空军学院）；The City College of New York（纽约市立学院）；Map Ltd（Map有限公司）
In this work, very deep super-resolution (VDSR) method is presented for improving the spatial resolution of remotely sensed (RS) images for scale factor 4. The VDSR net is re-trained with Sentinel-2 images and with drone aero orthophoto images, thus becomes RS-VDSR and Aero-VDSR, respectively. A novel loss function, the Var-norm estimator, is proposed in the regression layer of the convolutional neural network during re-training and prediction. According to numerical and optical comparisons, the proposed nets RS-VDSR and Aero-VDSR can outperform VDSR during prediction with RS images. RS-VDSR o…
In this paper, we present the optical image simulation from a synthetic aperture radar (SAR) data using deep learning based methods. Two models, i.e., optical image simulation directly from the SAR data and from multi-temporal SARoptical data, are proposed to testify the possibilities. The deep learning based methods that we chose to achieve the models are a convolutional neural network (CNN) with a residual architecture and a conditional generative adversarial network (cGAN). We validate our models using the Sentinel-1 and -2 datasets. The experiments demonstrate that the model with multi-tem…
Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院）
Deep Convolutional Neural Network (CNN) features have been demonstrated to be effective perceptual quality features. The perceptual loss, based on feature maps of pre-trained CNN's has proven to be remarkably effective for CNN based perceptual image restoration problems. In this work, taking inspiration from the the Human Visual System (HVS) and visual perception, we propose a spatial attention mechanism based on the dependency human contrast sensitivity on spatial frequency. We identify regions in input images, based on the underlying spatial frequency, which are not generally well reconstruc…
We propose a highly efficient and faster Single Image Super-Resolution (SISR) model with Deep Convolutional neural networks (Deep CNN). Deep CNN have recently shown that they have a significant reconstruction performance on single-image super-resolution. Current trend is using deeper CNN layers to improve performance. However, deep models demand larger computation resources and is not suitable for network edge devices like mobile, tablet and IoT devices. Our model achieves state of the art reconstruction performance with at least 10 times lower calculation cost by Deep CNN with Residual Net, S…
Hanyang University（汉阳大学）；LG AI Research（LG人工智能研究院）
Recent transformer-based super-resolution (SR) methods have achieved promising results against conventional CNN-based methods. However, these approaches suffer from essential shortsightedness created by only utilizing the standard self-attention-based reasoning. In this paper, we introduce an effective hybrid SR network to aggregate enriched features, including local features from CNNs and long-range multi-scale dependencies captured by transformers. Specifically, our network comprises transformer and convolutional branches, which synergetically complement each representation during the restor…
University of Salzburg（萨尔茨堡大学）；Federal University of Tocantins（托坎廷斯联邦大学）；Halmstad University（哈尔姆斯塔德大学）
The use of low-resolution images adopting more relaxed acquisition conditions such as mobile phones and surveillance videos is becoming increasingly common in iris recognition nowadays. Concurrently, a great variety of single image super-resolution techniques are emerging, especially with the use of convolutional neural networks (CNNs). The main objective of these methods is to try to recover finer texture details generating more photo-realistic images based on the optimisation of an objective function depending basically on the CNN architecture and training approach. In this work, the authors…
Recently, the example-based single image spectral reconstruction from RGB images task, aka, spectral super-resolution was approached by means of deep learning by Galliani et al. The proposed very deep convolutional neural network (CNN) achieved superior performance on recent large benchmarks. However, Aeschbacher et al showed that comparable performance can be achieved by shallow learning method based on A+, a method introduced for image super-resolution by Timofte et al. In this paper, we propose a moderately deep CNN model and substantially improve the reported performance on three spectral …
Wuhan Institute of Technology（武汉工程大学）；Harbin Institute of Technology（哈尔滨工业大学）；Wuhan University（武汉大学）
Recently, face super-resolution (FSR) methods either feed whole face image into convolutional neural networks (CNNs) or utilize extra facial priors (e.g., facial parsing maps, facial landmarks) to focus on facial structure, thereby maintaining the consistency of the facial structure while restoring facial details. However, the limited receptive fields of CNNs and inaccurate facial priors will reduce the naturalness and fidelity of the reconstructed face. In this paper, we propose a novel paradigm based on the self-attention mechanism (i.e., the core of Transformer) to fully explore the represe…
State of the art magnetic resonance (MR) image super-resolution methods (ISR) using convolutional neural networks (CNNs) leverage limited contextual information due to the limited spatial coverage of CNNs. Vision transformers (ViT) learn better global context that is helpful in generating superior quality HR images. We combine local information of CNNs and global information from ViTs for image super resolution and output super resolved images that have superior quality than those produced by state of the art methods. We include extra constraints through multiple novel loss functions that pres…
Multi-image super-resolution from multi-temporal satellite acquisitions of a scene has recently enjoyed great success thanks to new deep learning models. In this paper, we go beyond classic image reconstruction at a higher resolution by studying a super-resolved inference problem, namely semantic segmentation at a spatial resolution higher than the one of sensing platform. We expand upon recently proposed models exploiting temporal permutation invariance with a multi-resolution fusion module able to infer the rich semantic information needed by the segmentation task. The model presented in thi…
This paper presents the BigEarthNet that is a new large-scale multi-label Sentinel-2 benchmark archive. The BigEarthNet consists of 590,326 Sentinel-2 image patches, each of which is a section of i) 120x120 pixels for 10m bands; ii) 60x60 pixels for 20m bands; and iii) 20x20 pixels for 60m bands. Unlike most of the existing archives, each image patch is annotated by multiple land-cover classes (i.e., multi-labels) that are provided from the CORINE Land Cover database of the year 2018 (CLC 2018). The BigEarthNet is significantly larger than the existing archives in remote sensing (RS) and thus …
Technical University of Munich（慕尼黑工业大学）；German Aerospace Center（德国航空航天中心）；Aarhus University（奥胡斯大学）
Human settlement extent (HSE) information is a valuable indicator of world-wide urbanization as well as the resulting human pressure on the natural environment. Therefore, mapping HSE is critical for various environmental issues at local, regional, and even global scales. This paper presents a deep-learning-based framework to automatically map HSE from multi-spectral Sentinel-2 data using regionally available geo-products as training labels. A straightforward, simple, yet effective fully convolutional network-based architecture, Sen2HSE, is implemented as an example for semantic segmentation w…
Pacific Northwest National Laboratory（太平洋西北国家实验室）；Climate AI
Recently, deep Convolutional Neural Networks (CNNs) have revolutionized image super-resolution (SR), dramatically outperforming past methods for enhancing image resolution. They could be a boon for the many scientific fields that involve image or gridded datasets: satellite remote sensing, radar meteorology, medical imaging, numerical modeling etc. Unfortunately, while SR-CNNs produce visually compelling outputs, they may break physical conservation laws when applied to scientific datasets. Here, a method for ``Downsampling Enforcement" in SR-CNNs is proposed. A differentiable operator is deri…
Flow matching for Sentinel-2 super-resolution: implementation, application, and implications
Department of Agricultural and Biological Engineering, Mississippi State University（农业与生物工程系，密苏里州立大学）；North Mississippi Research and Extension Center, Mississippi State University（北密苏里研究与推广中心，密苏里州立大学）
Northwestern Polytechnical University（西北工业大学）；National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology（空天地海一体化大数据应用技术国家工程实验室）；Harbin Institute of Technology（哈尔滨工业大学）；National Tsing Hua University（国立清华大学）；The Chinese University of Hong Kong（香港中文大学）
Convolutional neural networks (CNNs) have obtained remarkable performance via deep architectures. However, these CNNs often achieve poor robustness for image super-resolution (SR) under complex scenes. In this paper, we present a heterogeneous group SR CNN (HGSRCNN) via leveraging structure information of different types to obtain a high-quality image. Specifically, each heterogeneous group block (HGB) of HGSRCNN uses a heterogeneous architecture containing a symmetric group convolutional block and a complementary convolutional block in a parallel way to enhance internal and external relations…
University of Florence（佛罗伦萨大学）；Centre Tecnològic Telecomunicacions Catalunya（加泰罗尼亚电信技术中心）；University of Padua（帕多瓦大学）
In this letter, we use deep-learning convolution neural networks (CNNs) to assess the landslide mapping and classification performances on optical images (from Sentinel-2) and SAR images (from Sentinel-1). The training and test zones used to independently evaluate the performance of the CNNs on different datasets are located in the eastern Iburi subprefecture in Hokkaido, where, at 03.08 local time (JST) on September 6, 2018, an Mw 6.6 earthquake triggered about 8000 coseismic landslides. We analyzed the conditions before and after the earthquake exploiting multi-polarization SAR as well as op…
Images play a vital role in understanding data through visual representation. It gives a clear representation of the object in context. But if this image is not clear it might not be of much use. Thus, the topic of Image Super Resolution arose and many researchers have been working towards applying Computer Vision and Deep Learning Techniques to increase the quality of images. One of the applications of Super Resolution is to increase the quality of Portrait Images. Portrait Images are images which mainly focus on capturing the essence of the main object in the frame, where the object in conte…
Sentinel-2 multi-spectral images collected over periods of several months were used to estimate vegetation height for Gabon and Switzerland. A deep convolutional neural network (CNN) was trained to extract suitable spectral and textural features from reflectance images and to regress per-pixel vegetation height. In Gabon, reference heights for training and validation were derived from airborne LiDAR measurements. In Switzerland, reference heights were taken from an existing canopy height model derived via photogrammetric surface reconstruction. The resulting maps have a mean absolute error (MA…
Wageningen University（瓦赫宁根大学）；Fondazione Bruno Kessler（布鲁诺·凯斯勒基金会）；Ecole Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）
While annotated images for change detection using satellite imagery are scarce and costly to obtain, there is a wealth of unlabeled images being generated every day. In order to leverage these data to learn an image representation more adequate for change detection, we explore methods that exploit the temporal consistency of Sentinel-2 times series to obtain a usable self-supervised learning signal. For this, we build and make publicly available (https://zenodo.org/record/4280482) the Sentinel-2 Multitemporal Cities Pairs (S2MTCP) dataset, containing multitemporal image pairs from 1520 urban a…
Super-resolution (SR) is a technique that allows increasing the resolution of a given image. Having applications in many areas, from medical imaging to consumer electronics, several SR methods have been proposed. Currently, the best performing methods are based on convolutional neural networks (CNNs) and require extensive datasets for training. However, at test time, they fail to impose consistency between the super-resolved image and the given low-resolution image, a property that classic reconstruction-based algorithms naturally enforce in spite of having poorer performance. Motivated by thi…
Single-image super-resolution is the process of increasing the resolution of an image, obtaining a high-resolution (HR) image from a low-resolution (LR) one. By leveraging large training datasets, convolutional neural networks (CNNs) currently achieve the state-of-the-art performance in this task. Yet, during testing/deployment, they fail to enforce consistency between the HR and LR images: if we downsample the output HR image, it never matches its LR input. Based on this observation, we propose to post-process the CNN outputs with an optimization problem that we call TV-TV minimization, which…
Télécom Paris, Institut Polytechnique de Paris（巴黎理工学院巴黎电信学院）；Laboratoire Hubert Curien（于贝尔·居里安实验室）
This paper presents a despeckling method for Sentinel-1 GRD images based on the recently proposed framework "SAR2SAR": a self-supervised training strategy. Training the deep neural network on collections of Sentinel 1 GRD images leads to a despeckling algorithm that is robust to space-variant spatial correlations of speckle. Despeckled images improve the detection of structures like narrow rivers. We apply a detector based on exogenous information and a linear features detector and show that rivers are better segmented when the processing chain is applied to images pre-processed by our despeck…
TU Kaiserslautern（凯撒斯劳滕工业大学）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心（DFKI））
In this paper, we address the challenge of land use and land cover classification using Sentinel-2 satellite images. The Sentinel-2 satellite images are openly and freely accessible provided in the Earth observation program Copernicus. We present a novel dataset based on Sentinel-2 satellite images covering 13 spectral bands and consisting out of 10 classes with in total 27,000 labeled and geo-referenced images. We provide benchmarks for this novel dataset with its spectral bands using state-of-the-art deep Convolutional Neural Network (CNNs). With the proposed novel dataset, we achieved an ov…
A new resolution enhancement method is presented for multispectral and multi-resolution images, such as these provided by the Sentinel-2 satellites. Starting from the highest resolution bands, band-dependent information (reflectance) is separated from information that is common to all bands (geometry of scene elements). This model is then applied to unmix low-resolution bands, preserving their reflectance, while propagating band-independent information to preserve the sub-pixel details. A reference implementation is provided, with an application example for super-resolving Sentinel-2 data.
This paper presents a generative adversarial network based super-resolution (SR) approach (which is called as S2GAN) to enhance the spatial resolution of Sentinel-2 spectral bands. The proposed approach consists of two main steps. The first step aims to increase the spatial resolution of the bands with 20m and 60m spatial resolutions by the scaling factors of 2 and 6, respectively. To this end, we introduce a generator network that performs SR on the lower resolution bands with the guidance of the bands associated to 10m spatial resolution by utilizing the convolutional layers with residual co…
Geographically-Weighted Weakly Supervised Bayesian High-Resolution Transformer for 200m Resolution Pan-Arctic Sea Ice Concentration Mapping and Uncertainty Estimation using Sentinel-1, RCM, and AMSR2 Data
Department of Geomatics Engineering, Schulich School of Engineering, University of Calgary（地质工程系，Schulich 工程学院，卡尔加里大学）
