Multispectral and hyperspectral imagery are widely used in agriculture, environmental monitoring, and urban planning due to their complementary spatial and spectral characteristics. A fundamental trade-off persists: multispectral imagery offers high spatial but limited spectral resolution, while hyperspectral imagery provides rich spectra at lower spatial resolution. Prior hyperspectral generation approaches (e.g., pan-sharpening variants, matrix factorization, CNNs) often struggle to jointly preserve spatial detail and spectral fidelity. In response, we propose SpecSwin3D, a transformer-based model that generates hyperspectral imagery from multispectral inputs while preserving both spatial and spectral quality. Specifically, SpecSwin3D takes five multispectral bands as input and reconstructs 224 hyperspectral bands at the same spatial resolution. In addition, we observe that reconstruction errors grow for hyperspectral bands spectrally distant from the input bands. To address this, we introduce a cascade training strategy that progressively expands the spectral range to stabilize learning and improve fidelity. Moreover, we design an optimized band sequence that strategically repeats and orders the five selected multispectral bands to better capture pairwise relations within a 3D shifted-window transformer framework. Quantitatively, our model achieves a PSNR of 35.82 dB, SAM of 2.40°, and SSIM of 0.96, outperforming the baseline MHF-Net by +5.6 dB in PSNR and reducing ERGAS by more than half. Beyond reconstruction, we further demonstrate the practical value of SpecSwin3D on two downstream tasks, including land use classification and burnt area segmentation.
In this paper, we propose a method using a three dimensional convolutional neural network (3-D-CNN) to fuse together multispectral (MS) and hyperspectral (HS) images to obtain a high resolution hyperspectral image. Dimensionality reduction of the hyperspectral image is performed prior to fusion in order to significantly reduce the computational time and make the method more robust to noise. Experiments are performed on a data set simulated using a real hyperspectral image. The results obtained show that the proposed approach is very promising when compared to conventional methods. This is espe…
German Aerospace Center (DLR)（德国航空航天中心）；Technical University of Munich (TUM)（慕尼黑工业大学）；RIKEN（理化学研究所）；Univ. Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；CNRS（法国国家科学研究中心）；Grenoble INP（格勒诺布尔国立理工学院）；University of Iceland（冰岛大学）
With a large amount of open satellite multispectral imagery (e.g., Sentinel-2 and Landsat-8), considerable attention has been paid to global multispectral land cover classification. However, its limited spectral information hinders further improving the classification performance. Hyperspectral imaging enables discrimination between spectrally similar classes but its swath width from space is narrow compared to multispectral ones. To achieve accurate land cover classification over a large coverage, we propose a cross-modality feature learning framework, called common subspace learning (CoSpace…
Spectral imaging enables spatially-resolved identification of materials in remote sensing, biomedicine, and astronomy. However, acquisition times require balancing spectral and spatial resolution with signal-to-noise. Hyperspectral imaging provides superior material specificity, while multispectral images are faster to collect at greater fidelity. We propose an approach for fusing hyperspectral and multispectral images to provide high-quality hyperspectral output. The proposed optimization leverages the least absolute shrinkage and selection operator (LASSO) to perform variable selection and r…
Fusing a low-resolution hyperspectral image (HSI) and a high-resolution multispectral image (MSI) of the same scene leads to a super-resolution image (SRI), which is information rich spatially and spectrally. In this paper, we super-resolve the HSI using the graph Laplacian defined on the MSI. Unlike many existing works, we don't assume prior knowledge about the spatial degradation from SRI to HSI, nor a perfectly aligned HSI and MSI pair. Our algorithm progressively alternates between finding the blur kernel and fusing HSI with MSI, generating accurate estimations of the blur kernel and the S…
Chester F. Carlson Center for Imaging Science Rochester Institute of Technology（切斯特·F·卡尔森成像科学中心 罗切斯特理工学院）
Deep convolutional neural networks (DCNNs) have been used to achieve state-of-the-art performance on many computer vision tasks (e.g., object recognition, object detection, semantic segmentation) thanks to a large repository of annotated image data. Large labeled datasets for other sensor modalities, e.g., multispectral imagery (MSI), are not available due to the large cost and manpower required. In this paper, we adapt state-of-the-art DCNN frameworks in computer vision for semantic segmentation for MSI imagery. To overcome label scarcity for MSI data, we substitute real MSI for generated syn…
Xi’an Jiaotong University（西安交通大学）；Harbin Institute of Technology（哈尔滨工业大学）
Hyperspectral imaging can help better understand the characteristics of different materials, compared with traditional image systems. However, only high-resolution multispectral (HrMS) and low-resolution hyperspectral (LrHS) images can generally be captured at video rate in practice. In this paper, we propose a model-based deep learning approach for merging an HrMS and LrHS images to generate a high-resolution hyperspectral (HrHS) image. In specific, we construct a novel MS/HS fusion model which takes the observation models of low-resolution images and the low-rankness knowledge along the spec…
Unmanned aircraft have decreased the cost required to collect remote sensing imagery, which has enabled researchers to collect high-spatial resolution data from multiple sensor modalities more frequently and easily. The increase in data will push the need for semantic segmentation frameworks that are able to classify non-RGB imagery, but this type of algorithmic development requires an increase in publicly available benchmark datasets with class labels. In this paper, we introduce a high-resolution multispectral dataset with image labels. This new benchmark dataset has been pre-split into trai…
Federal University of Santa Catarina（圣卡塔琳娜联邦大学）；Northeastern University（东北大学）；Catholic University of Pelotas（佩洛塔斯天主教大学）
Image fusion combines data from different heterogeneous sources to obtain more precise information about an underlying scene. Hyperspectral-multispectral (HS-MS) image fusion is currently attracting great interest in remote sensing since it allows the generation of high spatial resolution HS images, circumventing the main limitation of this imaging modality. Existing HS-MS fusion algorithms, however, neglect the spectral variability often existing between images acquired at different time instants. This time difference causes variations in spectral signatures of the underlying constituent mate…
Unsupervised Hyperspectral and Multispectral Image Blind Fusion Based on Deep Tucker Decomposition Network with Spatial-Spectral Manifold Learning
School of Computer Science and Engineering, Nanjing University of Science and Technology (NJUST)（南京理工大学计算机科学与工程学院）；Geological Exploration Technology Institute of Jiangsu Province（江苏省地质勘查技术院）；Jiangsu Province Engineering Research Center of Airborne Detecting and Intelligent Perceptive Technology（江苏省机载探测与智能感知技术工程研究中心）
In recent years, Hyperspectral Imaging (HSI) has become a powerful source for reliable data in applications such as remote sensing, agriculture, and biomedicine. However, hyperspectral images are highly data-dense and often benefit from methods to reduce the number of spectral bands while retaining the most useful information for a specific application. We propose a novel band selection method to select a reduced set of wavelengths, obtained from an HSI system in the context of image classification. Our approach consists of two main steps: the first utilizes a filter-based approach to find rel…
German Aerospace Center (DLR)（德国航空航天中心）；Technical University of Munich (TUM)（慕尼黑工业大学）；Univ. Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；Inria（法国国家信息与自动化研究所）；CNRS（法国国家科学研究中心）；Grenoble INP（格勒诺布尔国立理工学院）；University of Iceland（冰岛大学）；RIKEN（日本理化学研究所）
Due to the ever-growing diversity of the data source, multi-modality feature learning has attracted more and more attention. However, most of these methods are designed by jointly learning feature representation from multi-modalities that exist in both training and test sets, yet they are less investigated in absence of certain modality in the test phase. To this end, in this letter, we propose to learn a shared feature space across multi-modalities in the training process. By this way, the out-of-sample from any of multi-modalities can be directly projected onto the learned space for a more e…
Universite Bourgogne Franche-Comté（勃艮第-弗朗什-孔泰大学）；Le Centre de Recherche et de Restauration des Musées de France（法国博物馆研究与修复中心）；Centre National de la Recherche Scientifique（法国国家科学研究中心）
In the past few decades, there has been intensive research concerning the Unmixing of hyperspectral images. Some methods such as NMF, VCA, and N-FINDR have become standards since they show robustness in dealing with the unmixing of hyperspectral images. However, the research concerning the unmixing of multispectral images is relatively scarce. Thus, we extend some unmixing methods to the multispectral images. In this paper, we have created two simulated multispectral datasets from two hyperspectral datasets whose ground truths are given. Then we apply the unmixing methods (VCA, NMF, N-FINDR) t…
College of Electrical and Information Engineering, Hunan University（湖南大学电气与信息工程学院）
Hyperspectral image (HSI) with high spectral resolution often suffers from low spatial resolution owing to the limitations of imaging sensors. Image fusion is an effective and economical way to enhance the spatial resolution of HSI, which combines HSI with higher spatial resolution multispectral image (MSI) of the same scenario. In the past years, many HSI and MSI fusion algorithms are introduced to obtain high-resolution HSI. However, it lacks a full-scale review for the newly proposed HSI and MSI fusion approaches. To tackle this problem,this work gives a comprehensive review and new guideli…
Universidad Rey Juan Carlos（胡安·卡洛斯国王大学）；Universidad Industrial de Santander（桑坦德工业大学）
In remote sensing, hyperspectral (HS) and multispectral (MS) image fusion have emerged as a synthesis tool to improve the data set resolution. However, conventional image fusion methods typically degrade the performance of the land cover classification. In this paper, a feature fusion method from HS and MS images for pixel-based classification is proposed. More precisely, the proposed method first extracts spatial features from the MS image using morphological profiles. Then, the feature fusion model assumes that both the extracted morphological profiles and the HS image can be described as a …
University of Toulouse（图卢兹大学）；IRIT/INP-ENSEEIHT（图卢兹计算机研究所/国立图卢兹应用科学学院-国立高等电力技术、电子学、计算机、水力学与电信学院）；CNRS（法国国家科学研究中心）；Institut Universitaire de France (IUF)（法国大学研究院）；ISAE-SUPAERO（法国高等航空航天学院）；Institut de Recherche en Astrophysique et Planétologie (IRAP)（天体物理与行星学研究所）
Hyperspectral imaging has become a significant source of valuable data for astronomers over the past decades. Current instrumental and observing time constraints allow direct acquisition of multispectral images, with high spatial but low spectral resolution, and hyperspectral images, with low spatial but high spectral resolution. To enhance scientific interpretation of the data, we propose a data fusion method which combines the benefits of each image to recover a high spatio-spectral resolution datacube. The proposed inverse problem accounts for the specificities of astronomical instruments, …
University of South Carolina（南卡罗来纳大学）；East China Normal University（华东师范大学）
Nighttime satellite imagery has been applied in a wide range of fields. However, our limited understanding of how observed light intensity is formed and whether it can be simulated greatly hinders its further application. This study explores the potential of conditional Generative Adversarial Networks (cGAN) in translating multispectral imagery to nighttime imagery. A popular cGAN framework, pix2pix, was adopted and modified to facilitate this translation using gridded training image pairs derived from Landsat 8 and Visible Infrared Imaging Radiometer Suite (VIIRS). The results of this study p…
Center for Technologies in Robotics and Mechatronics Components（机器人与机电一体化组件技术中心）；Innopolis University（伊诺波利斯大学）；Institute of Robotics（机器人研究所）
Building detection from satellite multispectral imagery data is being a fundamental but a challenging problem mainly because it requires correct recovery of building footprints from high-resolution images. In this work, we propose a deep learning approach for building detection by applying numerous enhancements throughout the process. Initial dataset is preprocessed by 2-sigma percentile normalization. Then data preparation includes ensemble modelling where 3 models were created while incorporating OpenStreetMap data. Binary Distance Transformation (BDT) is used for improving data labeling pro…
School of Aerospace Engineering, Tsinghua University（清华大学航天航空学院）
Cross-modality fusing complementary information of multispectral remote sensing image pairs can improve the perception ability of detection algorithms, making them more robust and reliable for a wider range of applications, such as nighttime detection. Compared with prior methods, we think different features should be processed specifically, the modality-specific features should be retained and enhanced, while the modality-shared features should be cherry-picked from the RGB and thermal IR modalities. Following this idea, a novel and lightweight multispectral feature fusion approach with joint…
Implicit Neural Feature Fusion Function for Multispectral and Hyperspectral Image Fusion
University of Electronic Science and Technology of China（电子科技大学）；Institute of Methodologies for Environmental Analysis (CNR-IMAA)（环境分析研究所（CNR-IMAA））；National Biodiversity Future Center（国家生物多样性未来中心）
Multi-spectral satellite imagery provides valuable data at global scale for many environmental and socio-economic applications. Building supervised machine learning models based on these imagery, however, may require ground reference labels which are not available at global scale. Here, we propose a generative model to produce multi-resolution multi-spectral imagery based on Sentinel-2 data. The resulting synthetic images are indistinguishable from real ones by humans. This technique paves the road for future work to generate labeled synthetic imagery that can be used for data augmentation in …
ELTE Eötvös Loránd University（厄特沃什·罗兰大学（ELTE））；Tisza Plastic Cup Initiative（蒂萨塑料杯倡议组织）
One of the biggest environmental problems of our time is the increase in illegal landfills in forests, rivers, on river banks and other secluded places. In addition, waste in rivers causes damage not only locally, but also downstream, both in the water and washed ashore. Large islands of waste can also form at hydroelectric power stations and dams, and if they continue to flow, they can cause further damage to the natural environment along the river. Recent studies have also proved that rivers are the main source of plastic pollution in marine environments. Monitoring potential sources of dang…
California State Polytechnic University, Pomona（加州州立理工大学波莫纳分校）；University of California, Davis（加州大学戴维斯分校）
The large data size and dimensionality of hyperspectral data demands complex processing and data analysis. Multispectral data do not suffer the same limitations, but are normally restricted to blue, green, red, red edge, and near infrared bands. This study aimed to identify the optimal set of spectral bands for nitrogen detection in grape leaves using ensemble feature selection on hyperspectral data from over 3,000 leaves from 150 Flame Seedless table grapevines. Six machine learning base rankers were included in the ensemble: random forest, LASSO, SelectKBest, ReliefF, SVM-RFE, and chaotic cr…
In this paper, we propose a burnscar detection model for hyperspectral imaging (HSI) data. The proposed model contains two-processing steps in which the first step separate and then suppress the cloud information presenting in the data set using an RPCA algorithm and the second step detect the burnscar area in the low-rank component output of the first step. Experiments are conducted on the public MODIS dataset available at NASA official website.
Hyperspectral and multispectral image fusion with arbitrary resolution through self-supervised representations
Southern University of Science and Technology（南方科技大学）；The Chinese University of Hong Kong（香港中文大学）；University of Electronic Science and Technology of China（电子科技大学）；Hong Kong Baptist University（香港浸会大学）
Bayesian Fully-Connected Tensor Network for Hyperspectral-Multispectral Image Fusion
School of Computer Science and Technology, Huazhong University of Science and Technology（华中科技大学计算机科学与技术学院）；School of Computer Science and Artificial Intelligence, Zhengzhou University（郑州大学计算机科学与人工智能学院）；Department of Computer Science, St. Francis Xavier University（圣弗朗西斯大学计算机科学系）
Unmanned aerial vehicles (UAV) are used in precision agriculture (PA) to enable aerial monitoring of farmlands. Intelligent methods are required to pinpoint weed infestations and make optimal choice of pesticide. UAV can fly a multispectral camera and collect data. However, the classification of multispectral images using supervised machine learning algorithms such as convolutional neural networks (CNN) requires large amount of training data. This is a common drawback in deep learning we try to circumvent making use of a semi-supervised generative adversarial networks (GAN), providing a pixel-…
Gipsa-lab（Gipsa实验室）；ONERA（法国航空航天实验室）；Instituto de Telecomunicações（电信研究所）；Instituto Superior Técnico（高等技术学院）；Universidade de Lisboa（里斯本大学）；University of Toulouse（图卢兹大学）；Ghent University（根特大学）；North Atlantic Treaty Organization (NATO) Science and Technology Organization (STO) Centre for Maritime Research and Experimentation (CMRE)（北大西洋公约组织（NATO）科学与技术组织（STO）海事研究与实验中心（CMRE））；University of Tokyo（东京大学）
Pansharpening aims at fusing a panchromatic image with a multispectral one, to generate an image with the high spatial resolution of the former and the high spectral resolution of the latter. In the last decade, many algorithms have been presented in the literature for pansharpening using multispectral data. With the increasing availability of hyperspectral systems, these methods are now being adapted to hyperspectral images. In this work, we compare new pansharpening techniques designed for hyperspectral data with some of the state of the art methods for multispectral pansharpening, which hav…
In this paper a new method of image smoothing for satellite imagery and its applications in environmental remote sensing are presented. This method is based on the global gradient minimization over the whole image. With respect to the image discrete identity, the continuous minimization problem is discretized. Using the finite difference numerical method of differentiation, a simple yet efficient 5*5-pixel template is derived. Convolution of the derived template with the image in different bands results in the discrimination of various image elements. This method is extremely fast, besides bei…
ForCM: Forest Cover Mapping from Multispectral Sentinel-2 Image by Integrating Deep Learning with Object-Based Image Analysis
BRAC University（布拉克大学）；Charles Sturt University（查尔斯·斯特劳特大学）；University of New England（新英格兰大学）；University of Technology Sydney（技术悉尼大学）；Griffith University（格里菲斯大学）；Universiti Kebangsaan Malaysia (UKM)（马来亚大学）；Federation University Australia（联邦大学澳大利亚）；RMIT University（皇家墨尔本理工大学）
Evaluation of Deep Learning Semantic Segmentation for Land Cover Mapping on Multispectral, Hyperspectral and High Spatial Aerial Imagery
Center for Space and Remote Sensing Research, National Central University（国立中央大学太空及遥测研究中心）；Department of Computer Science and Information Engineering, National Central University（国立中央大学资讯工程学系）
This paper presents a structured dictionary-based model for hyperspectral data that incorporates both spectral and contextual characteristics of a spectral sample, with the goal of hyperspectral image classification. The idea is to partition the pixels of a hyperspectral image into a number of spatial neighborhoods called contextual groups and to model each pixel with a linear combination of a few dictionary elements learned from the data. Since pixels inside a contextual group are often made up of the same materials, their linear combinations are constrained to use common elements from the di…
University of California, Los Angeles（加州大学洛杉矶分校）；Bard College（巴德学院）；Grenoble Institute of Technology（格勒诺布尔理工学院）；University of California, San Diego（加州大学圣地亚哥分校）
In this paper, a graph-based nonlocal total variation method (NLTV) is proposed for unsupervised classification of hyperspectral images (HSI). The variational problem is solved by the primal-dual hybrid gradient (PDHG) algorithm. By squaring the labeling function and using a stable simplex clustering routine, an unsupervised clustering method with random initialization can be implemented. The effectiveness of this proposed algorithm is illustrated on both synthetic and real-world HSI, and numerical results show that the proposed algorithm outperforms other standard unsupervised clustering meth…
Taibah University（塔伊巴大学）；Donetsk National Technical University（顿涅茨克国立技术大学）
The paper presents a parallel implementation of existing image fusion methods on a graphical cluster. Parallel implementations of methods based on discrete wavelet transformation (Haars and Daubechies discrete wavelet transform) are developed. Experiments were performed on a cluster using GPU and CPU and performance gains were estimated for the use of the developed parallel implementations to process satellite images from satellite Landsat 7. The implementation on a graphic cluster provides performance improvement from 2 to 18 times. The quality of the considered methods was evaluated by ERGAS…
Fine-grained Hierarchical Crop Type Classification from Integrated Hyperspectral EnMAP Data and Multispectral Sentinel-2 Time Series: A Large-scale Dataset and Dual-stream Transformer Method
Jockey Club STEM Lab of Quantitative Remote Sensing, Department of Geography, The University of Hong Kong, Hong Kong, China（香港大学地理系定量遥感实验室）；Department of Aerospace Intelligent Science and Technology, School of Astronautics, Beihang University, Beijing, China（北航航天智能科学与技术系）
