Physically-Constrained Transfer Learning through Shared Abundance Space for Hyperspectral Image Classification
University of Tennessee, Knoxville（田纳西大学诺克斯维尔分校）；Beijing Institute of Technology（北京理工大学）；Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）
Hyperspectral image (HSI) classification is one of the most active research topics and has achieved promising results boosted by the recent development of deep learning. However, most state-of-the-art approaches tend to perform poorly when the training and testing images are on different domains, e.g., source domain and target domain, respectively, due to the spectral variability caused by different acquisition conditions. Transfer learning-based methods address this problem by pre-training in the source domain and fine-tuning on the target domain. Nonetheless, a considerable amount of data on the target domain has to be labeled and non-negligible computational resources are required to retrain the whole network. In this paper, we propose a new transfer learning scheme to bridge the gap between the source and target domains by projecting the HSI data from the source and target domains into a shared abundance space based on their own physical characteristics. In this way, the domain discrepancy would be largely reduced such that the model trained on the source domain could be applied on the target domain without extra efforts for data labeling or network retraining. The proposed method is referred to as physically-constrained transfer learning through shared abundance space (PCTL-SAS). Extensive experimental results demonstrate the superiority of the proposed method as compared to the state-of-the-art. The success of this endeavor would largely facilitate the deployment of HSI classification for real-world sensing scenarios.
Booz Allen Hamilton Inc.（博思艾伦汉密尔顿公司）；U.S. Army Research Laboratory（美国陆军研究实验室）
In this paper, we address the dataset scarcity issue with the hyperspectral image classification. As only a few thousands of pixels are available for training, it is difficult to effectively learn high-capacity Convolutional Neural Networks (CNNs). To cope with this problem, we propose a novel cross-domain CNN containing the shared parameters which can co-learn across multiple hyperspectral datasets. The network also contains the non-shared portions designed to handle the dataset specific spectral characteristics and the associated classification tasks. Our approach is the first attempt to lea…
University of Hong Kong（香港大学）；OneSpace Technology Co., Ltd.（零壹空间科技有限公司）；Sun Yat-sen University（中山大学）
In this letter, we propose a multitask deep learning method for classification of multiple hyperspectral data in a single training. Deep learning models have achieved promising results on hyperspectral image classification, but their performance highly rely on sufficient labeled samples, which are scarce on hyperspectral images. However, samples from multiple data sets might be sufficient to train one deep learning model, thereby improving its performance. To do so, we trained an identical feature extractor for all data, and the extracted features were fed into corresponding Softmax classifier…
Beijing Institute of Technology（北京理工大学）；The University of Hong Kong（香港大学）
Text information including extensive prior knowledge about land cover classes has been ignored in hyperspectral image classification (HSI) tasks. It is necessary to explore the effectiveness of linguistic mode in assisting HSI classification. In addition, the large-scale pre-training image-text foundation models have demonstrated great performance in a variety of downstream applications, including zero-shot transfer. However, most domain generalization methods have never addressed mining linguistic modal knowledge to improve the generalization performance of model. To compensate for the inadeq…
Army Research Laboratory（美国陆军研究实验室）；Booz Allen Hamilton Inc.（博思艾伦汉密尔顿公司）
A pretrain-finetune strategy is widely used to reduce the overfitting that can occur when data is insufficient for CNN training. First few layers of a CNN pretrained on a large-scale RGB dataset are capable of acquiring general image characteristics which are remarkably effective in tasks targeted for different RGB datasets. However, when it comes down to hyperspectral domain where each domain has its unique spectral properties, the pretrain-finetune strategy no longer can be deployed in a conventional way while presenting three major issues: 1) inconsistent spectral characteristics among the …
Real-time remote sensing applications like search and rescue missions, military target detection, environmental monitoring, hazard prevention and other time-critical applications require onboard real time processing capabilities or autonomous decision making. Some unmanned remote systems like satellites are physically remote from their operators, and all control of the spacecraft and data returned by the spacecraft must be transmitted over a wireless radio link. This link may not be available for extended periods when the satellite is out of line of sight of its ground station. Therefore, ligh…
German Aerospace Center (DLR)（德国航空航天中心）；Technical University of Munich (TUM)（慕尼黑工业大学）；Univ. Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；Inria（法国国家信息与自动化研究所）；CNRS（法国国家科学研究中心）；Grenoble INP（格勒诺布尔国立理工学院）；University of Iceland（冰岛大学）；RIKEN（日本理化学研究所）
Due to the ever-growing diversity of the data source, multi-modality feature learning has attracted more and more attention. However, most of these methods are designed by jointly learning feature representation from multi-modalities that exist in both training and test sets, yet they are less investigated in absence of certain modality in the test phase. To this end, in this letter, we propose to learn a shared feature space across multi-modalities in the training process. By this way, the out-of-sample from any of multi-modalities can be directly projected onto the learned space for a more e…
Dual-Branch Subpixel-Guided Network for Hyperspectral Image Classification
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；International Research Center of Big Data for Sustainable Development Goals（国际可持续发展大数据研究中心）；College of Resources and Environment, University of Chinese Academy of Sciences（中国科学院大学资源与环境学院）；Beijing Institute of Remote Sensing Equipment（北京遥感设备研究所）；Univ. Grenoble Alpes, CNRS, Grenoble INP, IJK（格勒诺布尔大学、法国国家科学研究中心、格勒诺布尔理工学院、图像与通信实验室）
Remote Sensing Technology Institute, German Aerospace Center（德国航空航天中心遥感技术研究所）；Technical University of Munich（慕尼黑工业大学）；Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；Univ. Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；INRIA（法国国家信息与自动化研究所）；CNRS（法国国家科学研究中心）；Grenoble INP（格勒诺布尔理工学院）
As remote sensing (RS) data obtained from different sensors become available largely and openly, multimodal data processing and analysis techniques have been garnering increasing interest in the RS and geoscience community. However, due to the gap between different modalities in terms of imaging sensors, resolutions, and contents, embedding their complementary information into a consistent, compact, accurate, and discriminative representation, to a great extent, remains challenging. To this end, we propose a shared and specific feature learning (S2FL) model. S2FL is capable of decomposing mult…
ONERA, The French Aerospace Lab（法国航空航天实验室（ONERA））；Univ. Bretagne-Sud（南布列塔尼大学）
This work addresses the scarcity of annotated hyperspectral data required to train deep neural networks. Especially, we investigate generative adversarial networks and their application to the synthesis of consistent labeled spectra. By training such networks on public datasets, we show that these models are not only able to capture the underlying distribution, but also to generate genuine-looking and physically plausible spectra. Moreover, we experimentally validate that the synthetic samples can be used as an effective data augmentation strategy. We validate our approach on several public hy…
Institute of Advanced Research in Artificial Intelligence（人工智能高级研究院）；Helmholtz-Zentrum Dresden-Rossendorf（亥姆霍兹德累斯顿罗森多夫研究中心）；Helmholtz Institute Freiberg for Resource Technology（亥姆霍兹弗赖贝格资源技术研究所）
In recent years, multi-view subspace learning has been garnering increasing attention. It aims to capture the inner relationships of the data that are collected from multiple sources by learning a unified representation. In this way, comprehensive information from multiple views is shared and preserved for the generalization processes. As a special branch of temporal series hyperspectral image (HSI) processing, the anomalous change detection task focuses on detecting very small changes among different temporal images. However, when the volume of datasets is very large or the classes are relati…
German Aerospace Center (DLR)（德国航空航天中心）；Technical University of Munich (TUM)（慕尼黑工业大学）；RIKEN（理化学研究所）；Univ. Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；CNRS（法国国家科学研究中心）；Grenoble INP（格勒诺布尔国立理工学院）；University of Iceland（冰岛大学）
With a large amount of open satellite multispectral imagery (e.g., Sentinel-2 and Landsat-8), considerable attention has been paid to global multispectral land cover classification. However, its limited spectral information hinders further improving the classification performance. Hyperspectral imaging enables discrimination between spectrally similar classes but its swath width from space is narrow compared to multispectral ones. To achieve accurate land cover classification over a large coverage, we propose a cross-modality feature learning framework, called common subspace learning (CoSpace…
Unsupervised Hyperspectral and Multispectral Image Blind Fusion Based on Deep Tucker Decomposition Network with Spatial-Spectral Manifold Learning
School of Computer Science and Engineering, Nanjing University of Science and Technology (NJUST)（南京理工大学计算机科学与工程学院）；Geological Exploration Technology Institute of Jiangsu Province（江苏省地质勘查技术院）；Jiangsu Province Engineering Research Center of Airborne Detecting and Intelligent Perceptive Technology（江苏省机载探测与智能感知技术工程研究中心）
Wuhan University（武汉大学）；Beijing Samsung Telecom R&D Center（北京三星通信技术研究中心）；University of Macau（澳门大学）
In hyperspectral remote sensing data mining, it is important to take into account of both spectral and spatial information, such as the spectral signature, texture feature and morphological property, to improve the performances, e.g., the image classification accuracy. In a feature representation point of view, a nature approach to handle this situation is to concatenate the spectral and spatial features into a single but high dimensional vector and then apply a certain dimension reduction technique directly on that concatenated vector before feed it into the subsequent classifier. However, mu…
We present a scalable low dimensional manifold model for the reconstruction of noisy and incomplete hyperspectral images. The model is based on the observation that the spatial-spectral blocks of a hyperspectral image typically lie close to a collection of low dimensional manifolds. To emphasize this, the dimension of the manifold is directly used as a regularizer in a variational functional, which is solved efficiently by alternating direction of minimization and weighted nonlocal Laplacian. Unlike general 3D images, the same similarity matrix can be shared across all spectral bands for a hyp…
Although many spectral unmixing models have been developed to address spectral variability caused by variable incident illuminations, the mechanism of the spectral variability is still unclear. This paper proposes an unmixing model, named illumination invariant spectral unmixing (IISU). IISU makes the first attempt to use the radiance hyperspectral data and a LiDAR-derived digital surface model (DSM) in order to physically explain variable illuminations and shadows in the unmixing framework. Incident angles, sky factors, visibility from the sun derived from the LiDAR-derived DSM support the ex…
Pontifical Catholic University of Rio de Janeiro（里约热内卢天主教大学）；IBM Research（IBM研究院）；McMaster University（麦克马斯特大学）；National Institute for Space Research (INPE)（巴西国家空间研究所）；Santa Catarina State University (UDESC)（圣卡塔琳娜州立大学）
This work proposes a multi-task fully convolutional architecture for tree species mapping in dense forests from sparse and scarce polygon-level annotations using hyperspectral UAV-borne data. Our model implements a partial loss function that enables dense tree semantic labeling outcomes from non-dense training samples, and a distance regression complementary task that enforces tree crown boundary constraints and substantially improves the model performance. Our multi-task architecture uses a shared backbone network that learns common representations for both tasks and two task-specific decoder…
The University of Tennessee（田纳西大学）；Applied Research LLC（应用研究有限责任公司）
In many computer vision applications, obtaining images of high resolution in both the spatial and spectral domains are equally important. However, due to hardware limitations, one can only expect to acquire images of high resolution in either the spatial or spectral domains. This paper focuses on hyperspectral image super-resolution (HSI-SR), where a hyperspectral image (HSI) with low spatial resolution (LR) but high spectral resolution is fused with a multispectral image (MSI) with high spatial resolution (HR) but low spectral resolution to obtain HR HSI. Existing deep learning-based solution…
Heterogeneous data fusion can enhance the robustness and accuracy of an algorithm on a given task. However, due to the difference in various modalities, aligning the sensors and embedding their information into discriminative and compact representations is challenging. In this paper, we propose a Contrastive learning based MultiModal Alignment Network (CoMMANet) to align data from different sensors into a shared and discriminative manifold where class information is preserved. The proposed architecture uses a multimodal triplet autoencoder to cluster the latent space in such a way that samples…
School of Marine Science and Technology, Northwestern Polytechnical University, China（西北工业大学海洋科学与技术学院）；State Key Laboratory of Rail Traffic Control and Safety, Beijing Jiaotong University, China（北京交通大学轨道交通控制与安全国家重点实验室）
Nowadays, nonnegative matrix factorization (NMF) based methods have been widely applied to blind spectral unmixing. Introducing proper regularizers to NMF is crucial for mathematically constraining the solutions and physically exploiting spectral and spatial properties of images. Generally, properly handcrafting regularizers and solving the associated complex optimization problem are non-trivial tasks. In our work, we propose an NMF based unmixing framework which jointly uses a handcrafting regularizer and a learnt regularizer from data. we plug learnt priors of abundances where the associated…
Hyperspectral imaging has the potential to improve intraoperative decision making if tissue characterisation is performed in real-time and with high-resolution. Hyperspectral snapshot mosaic sensors offer a promising approach due to their fast acquisition speed and compact size. However, a demosaicking algorithm is required to fully recover the spatial and spectral information of the snapshot images. Most state-of-the-art demosaicking algorithms require ground-truth training data with paired snapshot and high-resolution hyperspectral images, but such imagery pairs with the exact same scene are…
Physics-Informed Untrained Learning for RGB-Guided Superresolution Single-Pixel Hyperspectral Imaging
College of Science, Northwest Agriculture & Forestry University（西北农林科技大学理学院）；Key Laboratory of Photoelectronic Imaging Technology and System, School of Optics and Photonics, Beijing Institute of Technology（北京理工大学光电学院光电成像技术与系统教育部重点实验室）；State Key Laboratory of Digital Manufacturing Equipment and Technology, Huazhong University of Science and Technology（华中科技大学数字制造装备与技术国家重点实验室）
Harbin Institute of Technology（哈尔滨工业大学）；Wuhan University（武汉大学）
Recently, single gray/RGB image super-resolution reconstruction task has been extensively studied and made significant progress by leveraging the advanced machine learning techniques based on deep convolutional neural networks (DCNNs). However, there has been limited technical development focusing on single hyperspectral image super-resolution due to the high-dimensional and complex spectral patterns in hyperspectral image. In this paper, we make a step forward by investigating how to adapt state-of-the-art residual learning based single gray/RGB image super-resolution approaches for computati…
Physically-Induced Atmospheric Adversarial Perturbations: Enhancing Transferability and Robustness in Remote Sensing Image Classification
School of Computer and Information Engineering, Xiamen University of Technology（厦门理工学院计算机与信息工程学院）；Faculty of Data Science, City University of Macau（澳门城市大学数据科学学院）；Dundee International Institute, Central South University（中南大学 Dundee 国际学院）；Department of Electrical and Computer Engineering, University of Hong Kong（香港大学电气与计算机工程系）；Department of Engineering, University of Cambridge（剑桥大学工程系）；School of Computer Science, Engineering Research Center of Machine Learning and Industry Intelligence, Sichuan University（四川大学计算机科学学院、机器学习与工业智能工程研究中心）；Hong Kong JC STEM Lab of Smart City and Department of Computer Science, City University of Hong Kong（香港城市大学智慧城市JC STEM实验室与计算机科学系）；Department of Computer and Information Science, Faculty of Science and Technology, University of Macau（澳门大学科学与技术学院计算机与信息科学系）
Self-Supervised Super-Resolution for Sentinel-5P Hyperspectral Images
Université d’Orléans, Université de Tours, CNRS, IDP, UMR 7013（奥尔良大学、图尔大学、法国国家科学研究中心、IDP、UMR 7013）；Faculty of Mathematical Sciences, University of Khartoum（数学科学学院，喀土穆大学）；ENS Lyon, France（里昂高等师范学院，法国）；Laboratory of Physics and Chemistry of the Environment and Space (LPC2E), CNRS UMR 7328, University of Orléans, France（环境与空间物理化学实验室（LPC2E），法国国家科学研究中心UMR 7328，奥尔良大学，法国）；Institut universitaire de France (IUF), France（法国国家研究院（IUF），法国）
City University of Hong Kong（香港城市大学）；City University of Hong Kong Shenzhen Research Institute（香港城市大学深圳研究院）；Huaqiao University（华侨大学）；Xidian University（西安电子科技大学）；University of Macau（澳门大学）
In this paper, we investigate the problem of hyperspectral (HS) image spatial super-resolution via deep learning. Particularly, we focus on how to embed the high-dimensional spatial-spectral information of HS images efficiently and effectively. Specifically, in contrast to existing methods adopting empirically-designed network modules, we formulate HS embedding as an approximation of the posterior distribution of a set of carefully-defined HS embedding events, including layer-wise spatial-spectral feature extraction and network-level feature aggregation. Then, we incorporate the proposed featu…
Helmholtz-Zentrum Dresden-Rossendorf（亥姆霍兹德累斯顿-罗森多夫研究中心）；Univ. Grenoble Alpes（格勒诺布尔-阿尔卑斯大学）；CNRS（法国国家科学研究中心）；Grenoble INP（格勒诺布尔理工学院）；Nanjing University of Information Science and Technology（南京信息工程大学）；Hunan University（湖南大学）；Inria（法国国家信息与自动化研究所）；University of Iceland（冰岛大学）
Hyperspectral images provide detailed spectral information through hundreds of (narrow) spectral channels (also known as dimensionality or bands) with continuous spectral information that can accurately classify diverse materials of interest. The increased dimensionality of such data makes it possible to significantly improve data information content but provides a challenge to the conventional techniques (the so-called curse of dimensionality) for accurate analysis of hyperspectral images. Feature extraction, as a vibrant field of research in the hyperspectral community, evolved through decad…
University of Toulouse（图卢兹大学）；IRIT/INP-ENSEEIHT；Centre National d’Études Spatiales (CNES)（法国国家空间研究中心）；Institut Universitaire de France (IUF)（法国大学研究院）
Hyperspectral unmixing aims at identifying a set of elementary spectra and the corresponding mixture coefficients for each pixel of an image. As the elementary spectra correspond to the reflectance spectra of real materials, they are often very correlated yielding an ill-conditioned problem. To enrich the model and to reduce ambiguity due to the high correlation, it is common to introduce spatial information to complement the spectral information. The most common way to introduce spatial information is to rely on a spatial regularization of the abundance maps. In this paper, instead of conside…
Recently Convolutional Neural Networks (CNN) have been used to reconstruct hyperspectral information from RGB images. Moreover, this spectral reconstruction problem (SR) can often be solved with good (low) error. However, these methods are not physically plausible: that is when the recovered spectra are reintegrated with the underlying camera sensitivities, the resulting predicted RGB is not the same as the actual RGB, and sometimes this discrepancy can be large. The problem is further compounded by exposure change. Indeed, most learning-based SR models train for a fixed exposure setting and w…
SpectralEarth-FM: Bringing Hyperspectral Imagery into Multimodal Earth Observation Pretraining
Chair of Data Science in Earth Observation（地球观测数据科学主任）；Technical University of Munich（慕尼黑技术大学）；Remote Sensing Technology Institute（遥感技术研究所）；German Aerospace Center (DLR)（德国航空航天中心）；Department of Aerospace Engineering（航空航天工程系）；University of the Bundeswehr Munich（联邦国防军慕尼黑大学）；LEAP；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Univ. Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；Inria（法国国家信息与自动化技术研究院）；CNRS（法国国家科学研究中心）；Grenoble INP（格勒诺布尔INP）；LJK
SRL-SOA: Self-Representation Learning with Sparse 1D-Operational Autoencoder for Hyperspectral Image Band Selection
Faculty of Information Technology and Communication Sciences, Tampere University（坦佩雷大学信息技术与传播科学学院）；Department of Electrical Engineering, Qatar University（卡塔尔大学电气工程学院）；Tietoevry Oy（泰托埃弗里公司）
Degradation-Aware Metric Prompting for Hyperspectral Image Restoration
School of Computer Science and Technology, Beijing Institute of Technology, Beijing, China（北京理工大学计算机科学与技术学院）；School of Computer Science, Wuhan University, Wuhan, Hubei, China（武汉大学计算机学院）；Zhongguancun Academy, Beijing, China（中关村学院）
Zero-Shot Hyperspectral Pansharpening Using Hysteresis-Based Tuning for Spectral Quality Control
Department of Electrical Engineering and Information Technology, University Federico II（费米二世大学电气工程与信息科技系）；National Research Council, Institute of Methodologies for Environmental Analysis（国家研究 council 环境分析方法研究所）；Department of Engineering, University Parthenope（帕尔泰诺大学工程系）
Hyperspectral images are crucial for many research works. Spectral super-resolution (SSR) is a method used to obtain high spatial resolution (HR) hyperspectral images from HR multispectral images. Traditional SSR methods include model-driven algorithms and deep learning. By unfolding a variational method, this paper proposes an optimization-driven convolutional neural network (CNN) with a deep spatial-spectral prior, resulting in physically interpretable networks. Unlike the fully data-driven CNN, auxiliary spectral response function (SRF) is utilized to guide CNNs to group the bands with spec…
Deep Equilibrium Convolutional Sparse Coding for Hyperspectral Image Denoising
School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；College of Computer Science, Zhejiang University（浙江大学计算机科学学院）
Transforming Hyperspectral Images Into Chemical Maps: A Novel End-to-End Deep Learning Approach
FOSS Analytical A/S（福斯分析公司）；Department of Computer Science (DIKU), University of Copenhagen（哥本哈根大学计算机科学系）；Department of Food Science (UCPH FOOD), University of Copenhagen（哥本哈根大学食品科学系）；Information Technology Group, Wageningen University and Research (WUR)（瓦赫宁根大学及研究中心信息技术组）；Food and Biobased Research, Wageningen University and Research (WUR)（瓦赫宁根大学及研究中心食品与生物基研究）；Natural History Museum of Denmark (NHMD), University of Copenhagen（哥本哈根大学丹麦自然历史博物馆）
Couger Inc（库格公司）；Thiagarajar college of Engineering（蒂亚加拉贾工程学院）
Hyperspectral Imaging is the acquisition of spectral and spatial information of a particular scene. Capturing such information from a specialized hyperspectral camera remains costly. Reconstructing such information from the RGB image achieves a better solution in both classification and object recognition tasks. This work proposes a novel light weight network with very less number of parameters about 233,059 parameters based on Residual dense model with attention mechanism to obtain this solution. This network uses Coordination Convolutional Block to get the spatial information. The weights fr…
PhySIC: Physically Plausible 3D Human-Scene Interaction and Contact from a Single Image
University of Tübingen, Zuse School ELIZA（图宾根大学Zuse学校ELIZA）；University of Tübingen, Tübingen AI Center（图宾根大学图宾根人工智能中心）；University of Tübingen, Tübingen AI Center, MPI for Informatics, SIC（图宾根大学图宾根人工智能中心、马克斯·普朗克信息研究所、SIC）
Multi-modality (or multi-channel) imaging is becoming increasingly important and more widely available, e.g. hyperspectral imaging in remote sensing, spectral CT in material sciences as well as multi-contrast MRI and PET-MR in medicine. Research in the last decades resulted in a plethora of mathematical methods to combine data from several modalities. State-of-the-art methods, often formulated as variational regularization, have shown to significantly improve image reconstruction both quantitatively and qualitatively. Almost all of these models rely on the assumption that the modalities are pe…
It is time-consuming and expensive to take high-quality or high-resolution electron microscopy (EM) and fluorescence microscopy (FM) images. Taking these images could be even invasive to samples and may damage certain subtleties in the samples after long or intense exposures, often necessary for achieving high-quality or high resolution in the first place. Advances in deep learning enable us to perform image-to-image transformation tasks for various types of microscopy image reconstruction, computationally producing high-quality images from the physically acquired low-quality ones. When traini…
