Satellite imagery allows a plethora of applications ranging from weather forecasting to land surveying. The rapid development of computer vision systems could open new horizons to the utilization of satellite data due to the abundance of large volumes of data. However, current state-of-the-art computer vision systems mainly cater to applications that mainly involve natural images. While useful, those images exhibit a different distribution from satellite images in addition to having more spectral channels. This allows the use of pretrained deep learning models only in a subset of spectral channels that are equivalent to natural images thus discarding valuable information from other spectral channels. This calls for research effort to optimize deep learning models for satellite imagery to enable the assessment of their utility in the domain of remote sensing. Tensorflow tool allows for rapid prototyping and testing of deep learning models, however, its built-in image generator is designed to handle a maximum of four spectral channels. This manuscript introduces an open-source tool that allows the implementation of image augmentation for hyperspectral images in Tensorflow. Given how accessible and easy-to-use Tensorflow is, this tool would provide many researchers with the means to implement, test, and deploy deep learning models for remote sensing applications.
Herein, we present a system for hyperspectral image segmentation that utilizes multiple class--based denoising autoencoders which are efficiently trained. Moreover, we present a novel hyperspectral data augmentation method for labelled HSI data using linear mixtures of pixels from each class, which helps the system with edge pixels which are almost always mixed pixels. Finally, we utilize a deep neural network and morphological hole-filling to provide robust image classification. Results run on the Salinas dataset verify the high performance of the proposed algorithm.
Silesian University of Technology（西里西亚工业大学）；KP Labs（KP实验室）
Data augmentation is a popular technique which helps improve generalization capabilities of deep neural networks. It plays a pivotal role in remote-sensing scenarios in which the amount of high-quality ground truth data is limited, and acquiring new examples is costly or impossible. This is a common problem in hyperspectral imaging, where manual annotation of image data is difficult, expensive, and prone to human bias. In this letter, we propose online data augmentation of hyperspectral data which is executed during the inference rather than before the training of deep networks. This is in con…
Silesian University of Technology（西里西亚工业大学）；KP Labs（KP实验室）
Hyperspectral imaging provides detailed information about the scanned objects, as it captures their spectral characteristics within a large number of wavelength bands. Classification of such data has become an active research topic due to its wide applicability in a variety of fields. Deep learning has established the state of the art in the area, and it constitutes the current research mainstream. In this letter, we introduce a new spectral-spatial convolutional neural network, benefitting from a battery of data augmentation techniques which help deal with a real-life problem of lacking groun…
THAT: Token-wise High-frequency Augmentation Transformer for Hyperspectral Pansharpening
JPMorgan Chase（摩根大通公司）；University of Missouri-Kansas City（密苏里大学-堪萨斯城分校）；Ming Hsieh Department of Electrical and Computer Engineering（明希赫电气与计算机工程系）；University of Southern California（南加州大学）；Robinson Research Institute（罗宾逊研究学院）；KTH Royal Institute of Technology（皇家理工学院）；NEC Laboratories America（NEC美国实验室）；University of Virginia（弗吉尼亚大学）
Superpixel Graph Contrastive Clustering with Semantic-Invariant Augmentations for Hyperspectral Images
School of Software Engineering, Southeast University（东南大学软件学院）；School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China（新一代人工智能技术及交叉应用关键实验室（东南大学））；Department of Computing and Information Sciences, Saint Francis University（圣弗朗西斯大学计算机与信息科学系）；Department of Computer Science, City University of Hong Kong（香港城市大学计算机科学系）
Semantic segmentation of surgical hyperspectral images under geometric domain shifts
German Cancer Research Center (DKFZ)（德国癌症研究中心）；Heidelberg University（海德堡大学）；National Center for Tumor Diseases (NCT)（国家肿瘤疾病中心）；Heidelberg University Hospital（海德堡大学医院）
Kayrros SAS（Kayrros公司）；ENS Paris-Saclay（巴黎萨克雷高等师范学院）；University of Reims Champagne Ardenne（兰斯香槟阿登大学）；CNRS（法国国家科学研究中心）；Ecole Normale Supérieure（巴黎高等师范学院）
The new generation of hyperspectral imagers, such as PRISMA, has improved significantly our detection capability of methane (CH4) plumes from space at high spatial resolution (30m). We present here a complete framework to identify CH4 plumes using images from the PRISMA satellite mission and a deep learning model able to detect plumes over large areas. To compensate for the relative scarcity of PRISMA images, we trained our model by transposing high resolution plumes from Sentinel-2 to PRISMA. Our methodology thus avoids computationally expensive synthetic plume generation from Large Eddy Simu…
China University of Geosciences（中国地质大学）；Helmholtz-Zentrum Dresden-Rossendorf（德累斯顿罗森多夫亥姆霍兹中心）；Helmholtz Institute Freiberg for Resource Technology（弗赖贝格资源技术亥姆霍兹研究所）；Institute of Advanced Research in Artificial Intelligence（人工智能高级研究院）；National University of Defense Technology（国防科技大学）
Clustering of hyperspectral images is a fundamental but challenging task. The recent development of hyperspectral image clustering has evolved from shallow models to deep and achieved promising results in many benchmark datasets. However, their poor scalability, robustness, and generalization ability, mainly resulting from their offline clustering scenarios, greatly limit their application to large-scale hyperspectral data. To circumvent these problems, we present a scalable deep online clustering model, named Spectral-Spatial Contrastive Clustering (SSCC), based on self-supervised learning. S…
ONERA, The French Aerospace Lab（法国航空航天实验室（ONERA））；Univ. Bretagne-Sud（南布列塔尼大学）
This work addresses the scarcity of annotated hyperspectral data required to train deep neural networks. Especially, we investigate generative adversarial networks and their application to the synthesis of consistent labeled spectra. By training such networks on public datasets, we show that these models are not only able to capture the underlying distribution, but also to generate genuine-looking and physically plausible spectra. Moreover, we experimentally validate that the synthetic samples can be used as an effective data augmentation strategy. We validate our approach on several public hy…
Santa Clara University（圣克拉拉大学）；ETH Zürich（苏黎世联邦理工学院）；Westlake University（西湖大学）；Kuaishou Technology（快手科技）
Recently, hyperspectral imaging (HSI) has attracted increasing research attention, especially for the ones based on a coded aperture snapshot spectral imaging (CASSI) system. Existing deep HSI reconstruction models are generally trained on paired data to retrieve original signals upon 2D compressed measurements given by a particular optical hardware mask in CASSI, during which the mask largely impacts the reconstruction performance and could work as a "model hyperparameter" governing on data augmentations. This mask-specific training style will lead to a hardware miscalibration issue, which se…
Handling Geometric Domain Shifts in Semantic Segmentation of Surgical RGB and Hyperspectral Images
German Cancer Research Center (DKFZ)（德国癌症研究中心）；Helmholtz Information and Data Science School for Health（亥姆霍兹健康信息与数据科学学院）；Heidelberg University（海德堡大学）；National Center for Tumor Diseases (NCT)（国家肿瘤疾病中心）；Heidelberg University Hospital（海德堡大学医院）；Medical Faculty, Heidelberg University（海德堡大学医学院）
Friedrich-Alexander University Erlangen-Nuernberg（弗里德里希-亚历山大大学埃尔兰根-纽伦堡分校）；Gebze Technical University（盖布泽技术大学）；Sabanci University（萨班哲大学）
The amount of training data that is required to train a classifier scales with the dimensionality of the feature data. In hyperspectral remote sensing, feature data can potentially become very high dimensional. However, the amount of training data is oftentimes limited. Thus, one of the core challenges in hyperspectral remote sensing is how to perform multi-class classification using only relatively few training data points. In this work, we address this issue by enriching the feature matrix with synthetically generated sample points. This synthetic data is sampled from a GMM fitted to each cl…
Data augmentation is a critical component of training deep learning models. Although data augmentation has been shown to significantly improve image classification, its potential has not been thoroughly investigated for object detection. Given the additional cost for annotating images for object detection, data augmentation may be of even greater importance for this computer vision task. In this work, we study the impact of data augmentation on object detection. We first demonstrate that data augmentation operations borrowed from image classification may be helpful for training detection model…
Harbin Institute of Technology（哈尔滨工业大学）；Wuhan University（武汉大学）
High-resolution (HR) hyperspectral face image plays an important role in face related computer vision tasks under uncontrolled conditions, such as low-light environment and spoofing attacks. However, the dense spectral bands of hyperspectral face images come at the cost of limited amount of photons reached a narrow spectral window on average, which greatly reduces the spatial resolution of hyperspectral face images. In this paper, we investigate how to adapt the deep learning techniques to hyperspectral face image super-resolution (HFSR), especially when the training samples are very limited. …
CVL, ETH Zurich（苏黎世联邦理工学院计算机视觉实验室）；PSI, KU Leuven（鲁汶大学PSI实验室）
This work studies Hyperspectral image (HSI) super-resolution (SR). HSI SR is characterized by high-dimensional data and a limited amount of training examples. This exacerbates the undesirable behaviors of neural networks such as memorization and sensitivity to out-of-distribution samples. This work addresses these issues with three contributions. First, we observe that HSI SR and RGB image SR are correlated and develop a novel multi-tasking network to train them jointly so that the auxiliary task RGB image SR can provide additional supervision. Second, we propose a simple, yet effective data a…
Diffusion Models for Hyperspectral Image Analysis: A Comprehensive Review
School of Optical-Electrical and Computer Engineering, University of Shanghai for Science and Technology（上海理工大学光学电子与计算机工程学院）；School of Electronics and Electrical Engineering, Shanghai University of Engineering Science（上海工程技术大学电子与电气工程学院）；Medical Artificial Intelligence Lab, The First Hospital of Hebei Medical University, Hebei Medical University（河北医科大学第一医院医学人工智能实验室）；Hangzhou Institute of Technology, xidian University（杭州职业技术学院）
HSIGene: A Foundation Model For Hyperspectral Image Generation
School of Mathematics and Statistics, Northwestern Polytechnical University（西北工业大学数学与统计学院）；Xidian University（西安电子科技大学）；Pazhou Laboratory (Huangpu)（琶洲实验室（黄埔））
HIR-ALIGN: Enhancing Hyperspectral Image Restoration via Diffusion-Based Data Generation
School of Mathematics and Statistics, Xi’an Jiaotong University（西安交通大学数学与统计学学院）；School of Computer Science and Technology, Xi’an Jiaotong University（西安交通大学计算机科学与技术学院）；School of Mathematics and Statistics and the Ministry of Education Key Laboratory for Intelligent Networks and Network Security, Xi’an Jiaotong University（西安交通大学数学与统计学学院和教育部智能网络与网络安全重点实验室）；Pazhou Laboratory (Huangpu), Guangzhou（广州黄埔 Pazhou 实验室）
Revisiting Adversarial Training under Hyperspectral Image
School of Cyber Science and Engineering, Southeast University（东南大学网络空间安全学院）；Department of Computer Science, City University of Hong Kong（香港城市大学计算机科学系）；Engineering Research Center of Blockchain Application, Supervision and Management, Ministry of Education, Southeast University（东南大学教育部区块链应用监管工程研究中心）；Purple Mountain Laboratories（紫金山实验室）；Collaborative Innovation Center of Geospatial Technology, Wuhan University（武汉大学地理空间技术协同创新中心）；Department of Computer and Information Science, University of Macau（澳门大学计算机与信息科学系）；Department of Computer Science and Engineering, The Hong Kong University of Science and Technology（香港科技大学计算机科学与工程系）
Radboud University Nijmegen（拉德堡德大学奈梅亨分校）；Corbion（科碧恩）；Open University of the Netherlands（荷兰开放大学）
Spectral-spatial classification of hyperspectral images has been the subject of many studies in recent years. In the presence of only very few labeled pixels, this task becomes challenging. In this paper we address the following two research questions: 1) Can a simple neural network with just a single hidden layer achieve state of the art performance in the presence of few labeled pixels? 2) How is the performance of hyperspectral image classification methods affected when using disjoint train and test sets? We give a positive answer to the first question by using three tricks within a very ba…
MVOS_HSI: A Python Library for Preprocessing Agricultural Crop Hyperspectral Data
Machine Vision and Optical Sensor (MVOS) Lab, Dept. of Agricultural and Biosystems Engineering, South Dakota State University（农业与生物系统工程系机器视觉与光学传感器实验室，南达科他州立大学）；USDA/ARS Environmental Microbial and Food Safety Laboratory（美国农业部/动植物卫生检验局环境微生物与食品安全实验室）；Department of Agricultural and Biological Engineering, University of Florida（农业与生物工程系，佛罗里达大学）
Cross-Domain Few-Shot Learning for Hyperspectral Image Classification Based on Mixup Foundation Model
School of Computer Science and IT, Adelaide University（阿德莱德大学计算机科学与信息技术学院）；Department of Informatics, Institut Teknologi Sepuluh Nopember（泗水理工学院信息学系）；University of Technology, Sydney, Australia（悉尼科技大学）；Faculty of Computer Science, University of Indonesia（印度尼西亚大学计算机科学学院）
A Universal Knowledge Embedded Contrastive Learning Framework for Hyperspectral Image Classification
School of Resource and Environmental Sciences, Wuhan University（武汉大学资源与环境科学学院）；College of Science and Engineering, James Cook University（詹姆斯库克大学科学与工程学院）；School of Computer Science, Wuhan University（武汉大学计算机学院）
Institute of Information and Communication Technology, Bangladesh University of Engineering and Technology（孟加拉国工程技术大学信息与通信技术研究所）
Lung disease is common throughout the world. These include chronic obstructive pulmonary disease, pneumonia, asthma, tuberculosis, fibrosis, etc. Timely diagnosis of lung disease is essential. Many image processing and machine learning models have been developed for this purpose. Different forms of existing deep learning techniques including convolutional neural network (CNN), vanilla neural network, visual geometry group based neural network (VGG), and capsule network are applied for lung disease prediction.The basic CNN has poor performance for rotated, tilted, or other abnormal image orient…
AI-Driven HSI: Multimodality, Fusion, Challenges, and the Deep Learning Revolution
School of Electrical Engineering and Computer Science, Gwangju Institute of Science and Technology (GIST)（光州科学技术院电气工程与计算机科学学院）；Artificial Intelligence Graduate School, GIST（光州科学技术院人工智能研究生院）
University College London（伦敦大学学院）；National Hospital for Neurology and Neurosurgery（国家神经病学和神经外科医院）
Medical image analysis and computer-assisted intervention problems are increasingly being addressed with deep-learning-based solutions. Established deep-learning platforms are flexible but do not provide specific functionality for medical image analysis and adapting them for this application requires substantial implementation effort. Thus, there has been substantial duplication of effort and incompatible infrastructure developed across many research groups. This work presents the open-source NiftyNet platform for deep learning in medical imaging. The ambition of NiftyNet is to accelerate and …
Xeno-learning: knowledge transfer across species in deep learning-based spectral image analysis
Division of Intelligent Medical Systems (IMSY), German Cancer Research Center (DKFZ), Heidelberg, Germany（智能医疗系统部门，德国癌症研究中心（DKFZ），海德堡，德国）；Helmholtz Information and Data Science School for Health, Heidelberg/Karlsruhe, Germany（健康信息与数据科学学院，海德堡/卡尔斯鲁厄，德国）；Faculty of Mathematics and Computer Science, Heidelberg University, Heidelberg, Germany（数学与计算机科学学院，海德堡大学，海德堡，德国）；National Center for Tumor Diseases (NCT), NCT Heidelberg, a partnership between DKFZ and university medical center Heidelberg（肿瘤疾病国家研究中心（NCT），海德堡NCT，德国癌症研究中心（DKFZ）与海德堡大学医学中心的合作机构）；Department of General, Visceral, and Transplantation Surgery, Heidelberg University Hospital, Heidelberg, Germany（普通外科、 visceral外科及移植外科部门，海德堡大学医院，海德堡，德国）；Department of Urology and Urosurgery, Medical Faculty of the University of Heidelberg, University Medical Center Mannheim, Mannheim, Germany（泌尿外科与泌尿外科部门，海德堡大学医学院，曼海姆大学医学中心，曼海姆，德国）；Division of Intelligent Systems and Robotics in Urology (ISRU), German Cancer Research Center (DKFZ) Heidelberg, Heidelberg, Germany（泌尿外科智能系统与机器人部门（ISRU），德国癌症研究中心（DKFZ）海德堡，海德堡，德国）；DKFZ Hector Cancer Institute at the University Medical Center Mannheim, Mannheim, Germany（德国癌症研究中心（DKFZ）曼海姆癌症研究所，曼海姆，德国）；Division of Biostatistics, German Cancer Research Center (DKFZ), Heidelberg, Germany（生物统计学部门，德国癌症研究中心（DKFZ），海德堡，德国）；Heidelberg University, Medic（海德堡大学，Medic）
