Hyperspectral imaging provides detailed information about the scanned objects, as it captures their spectral characteristics within a large number of wavelength bands. Classification of such data has become an active research topic due to its wide applicability in a variety of fields. Deep learning has established the state of the art in the area, and it constitutes the current research mainstream. In this letter, we introduce a new spectral-spatial convolutional neural network, benefitting from a battery of data augmentation techniques which help deal with a real-life problem of lacking ground-truth training data. Our rigorous experiments showed that the proposed method outperforms other spectral-spatial techniques from the literature, and delivers precise hyperspectral classification in real time.
Silesian University of Technology（西里西亚科技大学）；KP Labs
Deep learning has established the state of the art in multiple fields, including hyperspectral image analysis. However, training large-capacity learners to segment such imagery requires representative training sets. Acquiring such data is human-dependent and time-consuming, especially in Earth observation scenarios, where the hyperspectral data transfer is very costly and time-constrained. In this letter, we show how to effectively deal with a limited number and size of available hyperspectral ground-truth sets, and apply transfer learning for building deep feature extractors. Also, we exploit…
Silesian University of Technology（西里西亚工业大学）；KP Labs（KP实验室）
Data augmentation is a popular technique which helps improve generalization capabilities of deep neural networks. It plays a pivotal role in remote-sensing scenarios in which the amount of high-quality ground truth data is limited, and acquiring new examples is costly or impossible. This is a common problem in hyperspectral imaging, where manual annotation of image data is difficult, expensive, and prone to human bias. In this letter, we propose online data augmentation of hyperspectral data which is executed during the inference rather than before the training of deep networks. This is in con…
Herein, we present a system for hyperspectral image segmentation that utilizes multiple class--based denoising autoencoders which are efficiently trained. Moreover, we present a novel hyperspectral data augmentation method for labelled HSI data using linear mixtures of pixels from each class, which helps the system with edge pixels which are almost always mixed pixels. Finally, we utilize a deep neural network and morphological hole-filling to provide robust image classification. Results run on the Salinas dataset verify the high performance of the proposed algorithm.
Superpixel Graph Contrastive Clustering with Semantic-Invariant Augmentations for Hyperspectral Images
School of Software Engineering, Southeast University（东南大学软件学院）；School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China（新一代人工智能技术及交叉应用关键实验室（东南大学））；Department of Computing and Information Sciences, Saint Francis University（圣弗朗西斯大学计算机与信息科学系）；Department of Computer Science, City University of Hong Kong（香港城市大学计算机科学系）
The problem of unsupervised learning and segmentation of hyperspectral images is a significant challenge in remote sensing. The high dimensionality of hyperspectral data, presence of substantial noise, and overlap of classes all contribute to the difficulty of automatically clustering and segmenting hyperspectral images. We propose an unsupervised learning technique called spectral-spatial diffusion learning (DLSS) that combines a geometric estimation of class modes with a diffusion-inspired labeling that incorporates both spectral and spatial information. The mode estimation incorporates the …
Center for Computing Research, Sandia National Laboratories（桑迪亚国家实验室计算研究中心）
The high dimensionality of hyperspectral imaging forces unique challenges in scope, size and processing requirements. Motivated by the potential for an in-the-field cell sorting detector, we examine a $\textit{Synechocystis sp.}$ PCC 6803 dataset wherein cells are grown alternatively in nitrogen rich or deplete cultures. We use deep learning techniques to both successfully classify cells and generate a mask segmenting the cells/condition from the background. Further, we use the classification accuracy to guide a data-driven, iterative feature selection method, allowing the design neural networ…
Addressing Annotation Scarcity in Hyperspectral Brain Image Segmentation with Unsupervised Domain Adaptation
Technical University of Munich, Munich, Germany（慕尼黑技术大学）；Department of Computing, Imperial College London, London, UK（伦敦帝国学院计算机系）；Munich Center for Machine Learning (MCML), Munich, Germany（慕尼黑机器学习中心（MCML））
Magnetic resonance images (MRIs) are widely used to quantify vestibular schwannoma and the cochlea. Recently, deep learning methods have shown state-of-the-art performance for segmenting these structures. However, training segmentation models may require manual labels in target domain, which is expensive and time-consuming. To overcome this problem, domain adaptation is an effective way to leverage information from source domain to obtain accurate segmentations without requiring manual labels in target domain. In this paper, we propose an unsupervised learning framework to segment the VS and c…
Satellite imagery allows a plethora of applications ranging from weather forecasting to land surveying. The rapid development of computer vision systems could open new horizons to the utilization of satellite data due to the abundance of large volumes of data. However, current state-of-the-art computer vision systems mainly cater to applications that mainly involve natural images. While useful, those images exhibit a different distribution from satellite images in addition to having more spectral channels. This allows the use of pretrained deep learning models only in a subset of spectral chan…
THAT: Token-wise High-frequency Augmentation Transformer for Hyperspectral Pansharpening
JPMorgan Chase（摩根大通公司）；University of Missouri-Kansas City（密苏里大学-堪萨斯城分校）；Ming Hsieh Department of Electrical and Computer Engineering（明希赫电气与计算机工程系）；University of Southern California（南加州大学）；Robinson Research Institute（罗宾逊研究学院）；KTH Royal Institute of Technology（皇家理工学院）；NEC Laboratories America（NEC美国实验室）；University of Virginia（弗吉尼亚大学）
Huazhong University of Science and Technology（华中科技大学）；Kennesaw State University（肯尼索州立大学）
The crossMoDA challenge aims to automatically segment the vestibular schwannoma (VS) tumor and cochlea regions of unlabeled high-resolution T2 scans by leveraging labeled contrast-enhanced T1 scans. The 2022 edition extends the segmentation task by including multi-institutional scans. In this work, we proposed an unpaired cross-modality segmentation framework using data augmentation and hybrid convolutional networks. Considering heterogeneous distributions and various image sizes for multi-institutional scans, we apply the min-max normalization for scaling the intensities of all scans between …
Friedrich-Alexander University Erlangen-Nuernberg（弗里德里希-亚历山大大学埃尔兰根-纽伦堡分校）；Gebze Technical University（盖布泽技术大学）；Sabanci University（萨班哲大学）
The amount of training data that is required to train a classifier scales with the dimensionality of the feature data. In hyperspectral remote sensing, feature data can potentially become very high dimensional. However, the amount of training data is oftentimes limited. Thus, one of the core challenges in hyperspectral remote sensing is how to perform multi-class classification using only relatively few training data points. In this work, we address this issue by enriching the feature matrix with synthetically generated sample points. This synthetic data is sampled from a GMM fitted to each cl…
Harbin Institute of Technology（哈尔滨工业大学）；Wuhan University（武汉大学）
High-resolution (HR) hyperspectral face image plays an important role in face related computer vision tasks under uncontrolled conditions, such as low-light environment and spoofing attacks. However, the dense spectral bands of hyperspectral face images come at the cost of limited amount of photons reached a narrow spectral window on average, which greatly reduces the spatial resolution of hyperspectral face images. In this paper, we investigate how to adapt the deep learning techniques to hyperspectral face image super-resolution (HFSR), especially when the training samples are very limited. …
DIKU University of Copenhagen（哥本哈根大学DIKU）；Biomediq A/S（Biomediq公司）；Cerebriu A/S（Cerebriu公司）；King’s College London（伦敦国王学院）；University College London（伦敦大学学院）
For proper generalization performance of convolutional neural networks (CNNs) in medical image segmentation, the learnt features should be invariant under particular non-linear shape variations of the input. To induce invariance in CNNs to such transformations, we propose Probabilistic Augmentation of Data using Diffeomorphic Image Transformation (PADDIT) -- a systematic framework for generating realistic transformations that can be used to augment data for training CNNs. We show that CNNs trained with PADDIT outperforms CNNs trained without augmentation and with generic augmentation in segmen…
Australian Centre for Field Robotics（澳大利亚野外机器人中心）；University of Sydney（悉尼大学）
Hyperspectral imaging sensors are becoming increasingly popular in robotics applications such as agriculture and mining, and allow per-pixel thematic classification of materials in a scene based on their unique spectral signatures. Recently, convolutional neural networks have shown remarkable performance for classification tasks, but require substantial amounts of labelled training data. This data must sufficiently cover the variability expected to be encountered in the environment. For hyperspectral data, one of the main variations encountered outdoors is due to incident illumination, which c…
ONERA, The French Aerospace Lab（法国航空航天实验室（ONERA））；Univ. Bretagne-Sud（南布列塔尼大学）
This work addresses the scarcity of annotated hyperspectral data required to train deep neural networks. Especially, we investigate generative adversarial networks and their application to the synthesis of consistent labeled spectra. By training such networks on public datasets, we show that these models are not only able to capture the underlying distribution, but also to generate genuine-looking and physically plausible spectra. Moreover, we experimentally validate that the synthetic samples can be used as an effective data augmentation strategy. We validate our approach on several public hy…
CVL, ETH Zurich（苏黎世联邦理工学院计算机视觉实验室）；PSI, KU Leuven（鲁汶大学PSI实验室）
This work studies Hyperspectral image (HSI) super-resolution (SR). HSI SR is characterized by high-dimensional data and a limited amount of training examples. This exacerbates the undesirable behaviors of neural networks such as memorization and sensitivity to out-of-distribution samples. This work addresses these issues with three contributions. First, we observe that HSI SR and RGB image SR are correlated and develop a novel multi-tasking network to train them jointly so that the auxiliary task RGB image SR can provide additional supervision. Second, we propose a simple, yet effective data a…
Kayrros SAS（Kayrros公司）；ENS Paris-Saclay（巴黎萨克雷高等师范学院）；University of Reims Champagne Ardenne（兰斯香槟阿登大学）；CNRS（法国国家科学研究中心）；Ecole Normale Supérieure（巴黎高等师范学院）
The new generation of hyperspectral imagers, such as PRISMA, has improved significantly our detection capability of methane (CH4) plumes from space at high spatial resolution (30m). We present here a complete framework to identify CH4 plumes using images from the PRISMA satellite mission and a deep learning model able to detect plumes over large areas. To compensate for the relative scarcity of PRISMA images, we trained our model by transposing high resolution plumes from Sentinel-2 to PRISMA. Our methodology thus avoids computationally expensive synthetic plume generation from Large Eddy Simu…
HIR-ALIGN: Enhancing Hyperspectral Image Restoration via Diffusion-Based Data Generation
School of Mathematics and Statistics, Xi’an Jiaotong University（西安交通大学数学与统计学学院）；School of Computer Science and Technology, Xi’an Jiaotong University（西安交通大学计算机科学与技术学院）；School of Mathematics and Statistics and the Ministry of Education Key Laboratory for Intelligent Networks and Network Security, Xi’an Jiaotong University（西安交通大学数学与统计学学院和教育部智能网络与网络安全重点实验室）；Pazhou Laboratory (Huangpu), Guangzhou（广州黄埔 Pazhou 实验室）
Revisiting Adversarial Training under Hyperspectral Image
School of Cyber Science and Engineering, Southeast University（东南大学网络空间安全学院）；Department of Computer Science, City University of Hong Kong（香港城市大学计算机科学系）；Engineering Research Center of Blockchain Application, Supervision and Management, Ministry of Education, Southeast University（东南大学教育部区块链应用监管工程研究中心）；Purple Mountain Laboratories（紫金山实验室）；Collaborative Innovation Center of Geospatial Technology, Wuhan University（武汉大学地理空间技术协同创新中心）；Department of Computer and Information Science, University of Macau（澳门大学计算机与信息科学系）；Department of Computer Science and Engineering, The Hong Kong University of Science and Technology（香港科技大学计算机科学与工程系）
Radboud University Nijmegen（拉德堡德大学奈梅亨分校）；Corbion（科碧恩）；Open University of the Netherlands（荷兰开放大学）
Spectral-spatial classification of hyperspectral images has been the subject of many studies in recent years. In the presence of only very few labeled pixels, this task becomes challenging. In this paper we address the following two research questions: 1) Can a simple neural network with just a single hidden layer achieve state of the art performance in the presence of few labeled pixels? 2) How is the performance of hyperspectral image classification methods affected when using disjoint train and test sets? We give a positive answer to the first question by using three tricks within a very ba…
China University of Geosciences（中国地质大学）；Helmholtz-Zentrum Dresden-Rossendorf（德累斯顿罗森多夫亥姆霍兹中心）；Helmholtz Institute Freiberg for Resource Technology（弗赖贝格资源技术亥姆霍兹研究所）；Institute of Advanced Research in Artificial Intelligence（人工智能高级研究院）；National University of Defense Technology（国防科技大学）
Clustering of hyperspectral images is a fundamental but challenging task. The recent development of hyperspectral image clustering has evolved from shallow models to deep and achieved promising results in many benchmark datasets. However, their poor scalability, robustness, and generalization ability, mainly resulting from their offline clustering scenarios, greatly limit their application to large-scale hyperspectral data. To circumvent these problems, we present a scalable deep online clustering model, named Spectral-Spatial Contrastive Clustering (SSCC), based on self-supervised learning. S…
Plant phenotyping tasks such as leaf segmentation and counting are fundamental to the study of phenotypic traits. Since it is well-suited for these tasks, deep supervised learning has been prevalent in recent works proposing better performing models at segmenting and counting leaves. Despite good efforts from research groups, one of the main challenges for proposing better methods is still the limitation of labelled data availability. The main efforts of the field seem to be augmenting existing limited data sets, and some aspects of the modelling process have been under-discussed. This paper e…
Image segmentation is an important task in many medical applications. Methods based on convolutional neural networks attain state-of-the-art accuracy; however, they typically rely on supervised training with large labeled datasets. Labeling medical images requires significant expertise and time, and typical hand-tuned approaches for data augmentation fail to capture the complex variations in such images. We present an automated data augmentation method for synthesizing labeled medical images. We demonstrate our method on the task of segmenting magnetic resonance imaging (MRI) brain scans. Our …
A Universal Knowledge Embedded Contrastive Learning Framework for Hyperspectral Image Classification
School of Resource and Environmental Sciences, Wuhan University（武汉大学资源与环境科学学院）；College of Science and Engineering, James Cook University（詹姆斯库克大学科学与工程学院）；School of Computer Science, Wuhan University（武汉大学计算机学院）
Semantic segmentation of surgical hyperspectral images under geometric domain shifts
German Cancer Research Center (DKFZ)（德国癌症研究中心）；Heidelberg University（海德堡大学）；National Center for Tumor Diseases (NCT)（国家肿瘤疾病中心）；Heidelberg University Hospital（海德堡大学医院）
Handling Geometric Domain Shifts in Semantic Segmentation of Surgical RGB and Hyperspectral Images
German Cancer Research Center (DKFZ)（德国癌症研究中心）；Helmholtz Information and Data Science School for Health（亥姆霍兹健康信息与数据科学学院）；Heidelberg University（海德堡大学）；National Center for Tumor Diseases (NCT)（国家肿瘤疾病中心）；Heidelberg University Hospital（海德堡大学医院）；Medical Faculty, Heidelberg University（海德堡大学医学院）
Medical Image Segmentation is a useful application for medical image analysis including detecting diseases and abnormalities in imaging modalities such as MRI, CT etc. Deep learning has proven to be promising for this task but usually has a low accuracy because of the lack of appropriate publicly available annotated or segmented medical datasets. In addition, the datasets that are available may have a different texture because of different dosage values or scanner properties than the images that need to be segmented. This paper presents a StyleGAN-driven approach for segmenting publicly availa…
Inria（法国国家信息与自动化研究所）；IHU Liryc（利里克大学医院研究所）；University of Bordeaux（波尔多大学）
Radiological imaging offers effective measurement of anatomy, which is useful in disease diagnosis and assessment. Previous study has shown that the left atrial wall remodeling can provide information to predict treatment outcome in atrial fibrillation. Nevertheless, the segmentation of the left atrial structures from medical images is still very time-consuming. Current advances in neural network may help creating automatic segmentation models that reduce the workload for clinicians. In this preliminary study, we propose automated, two-stage, three-dimensional U-Nets with convolutional neural …
Gwangju Institute of Science and Technology (GIST)（光州科学技术院）
Understanding assembly instruction has the potential to enhance the robot s task planning ability and enables advanced robotic applications. To recognize the key components from the 2D assembly instruction image, We mainly focus on segmenting the speech bubble area, which contains lots of information about instructions. For this, We applied Cascade Mask R-CNN and developed a context-aware data augmentation scheme for speech bubble segmentation, which randomly combines images cuts by considering the context of assembly instructions. We showed that the proposed augmentation scheme achieves a bet…
Ocean University of China（中国海洋大学）；Southwest Jiaotong University（西南交通大学）；Mississippi State University（密西西比州立大学）
The joint hyperspectral image (HSI) and LiDAR data classification aims to interpret ground objects at more detailed and precise level. Although deep learning methods have shown remarkable success in the multisource data classification task, self-supervised learning has rarely been explored. It is commonly nontrivial to build a robust self-supervised learning model for multisource data classification, due to the fact that the semantic similarities of neighborhood regions are not exploited in existing contrastive learning framework. Furthermore, the heterogeneous gap induced by the inconsistent …
RWTH Aachen University（亚琛工业大学）；NVIDIA（英伟达）；ETH AI Center（苏黎世联邦理工学院AI中心）
We present Mix3D, a data augmentation technique for segmenting large-scale 3D scenes. Since scene context helps reasoning about object semantics, current works focus on models with large capacity and receptive fields that can fully capture the global context of an input 3D scene. However, strong contextual priors can have detrimental implications like mistaking a pedestrian crossing the street for a car. In this work, we focus on the importance of balancing global scene context and local geometry, with the goal of generalizing beyond the contextual priors in the training set. In particular, we…
MVOS_HSI: A Python Library for Preprocessing Agricultural Crop Hyperspectral Data
Machine Vision and Optical Sensor (MVOS) Lab, Dept. of Agricultural and Biosystems Engineering, South Dakota State University（农业与生物系统工程系机器视觉与光学传感器实验室，南达科他州立大学）；USDA/ARS Environmental Microbial and Food Safety Laboratory（美国农业部/动植物卫生检验局环境微生物与食品安全实验室）；Department of Agricultural and Biological Engineering, University of Florida（农业与生物工程系，佛罗里达大学）
Diffusion Models for Hyperspectral Image Analysis: A Comprehensive Review
School of Optical-Electrical and Computer Engineering, University of Shanghai for Science and Technology（上海理工大学光学电子与计算机工程学院）；School of Electronics and Electrical Engineering, Shanghai University of Engineering Science（上海工程技术大学电子与电气工程学院）；Medical Artificial Intelligence Lab, The First Hospital of Hebei Medical University, Hebei Medical University（河北医科大学第一医院医学人工智能实验室）；Hangzhou Institute of Technology, xidian University（杭州职业技术学院）
Santa Clara University（圣克拉拉大学）；ETH Zürich（苏黎世联邦理工学院）；Westlake University（西湖大学）；Kuaishou Technology（快手科技）
Recently, hyperspectral imaging (HSI) has attracted increasing research attention, especially for the ones based on a coded aperture snapshot spectral imaging (CASSI) system. Existing deep HSI reconstruction models are generally trained on paired data to retrieve original signals upon 2D compressed measurements given by a particular optical hardware mask in CASSI, during which the mask largely impacts the reconstruction performance and could work as a "model hyperparameter" governing on data augmentations. This mask-specific training style will lead to a hardware miscalibration issue, which se…
HSIGene: A Foundation Model For Hyperspectral Image Generation
School of Mathematics and Statistics, Northwestern Polytechnical University（西北工业大学数学与统计学院）；Xidian University（西安电子科技大学）；Pazhou Laboratory (Huangpu)（琶洲实验室（黄埔））
