Hyperspectral satellite imaging attracts enormous research attention in the remote sensing community, hence automated approaches for precise segmentation of such imagery are being rapidly developed. In this letter, we share our observations on the strategy for validating hyperspectral image segmentation algorithms currently followed in the literature, and show that it can lead to over-optimistic experimental insights. We introduce a new routine for generating segmentation benchmarks, and use it to elaborate ready-to-use hyperspectral training-test data partitions. They can be utilized for fair validation of new and existing algorithms without any training-test data leakage.
Most semantic segmentation approaches of Hyperspectral images (HSIs) use and require preprocessing steps in the form of patching to accurately classify diversified land cover in remotely sensed images. These approaches use patching to incorporate the rich neighborhood information in images and exploit the simplicity and segmentability of the most common HSI datasets. In contrast, most landmasses in the world consist of overlapping and diffused classes, making neighborhood information weaker than what is seen in common HSI datasets. To combat this issue and generalize the segmentation models to…
Tufts University（塔夫茨大学）；City University of Hong Kong（香港城市大学）；Wake Forest University（维克森林大学）
Hyperspectral images encode rich structure that can be exploited for material discrimination by machine learning algorithms. This article introduces the Active Diffusion and VCA-Assisted Image Segmentation (ADVIS) for active material discrimination. ADVIS selects high-purity, high-density pixels that are far in diffusion distance (a data-dependent metric) from other high-purity, high-density pixels in the hyperspectral image. The ground truth labels of these pixels are queried and propagated to the rest of the image. The ADVIS active learning algorithm is shown to strongly outperform its fully…
CSIRO’s Data61（澳大利亚联邦科学与工业研究组织Data61）；Australian National University（澳大利亚国立大学）
Hyperspectral Imaging (HSI) provides detailed spectral information and has been utilised in many real-world applications. This work introduces an HSI dataset of building facades in a light industry environment with the aim of classifying different building materials in a scene. The dataset is called the Light Industrial Building HSI (LIB-HSI) dataset. This dataset consists of nine categories and 44 classes. In this study, we investigated deep learning based semantic segmentation algorithms on RGB and hyperspectral images to classify various building materials, such as timber, brick and concret…
We present a supervised hyperspectral image segmentation algorithm based on a convex formulation of a marginal maximum a posteriori segmentation with hidden fields and structure tensor regularization: Segmentation via the Constraint Split Augmented Lagrangian Shrinkage by Structure Tensor Regularization (SegSALSA-STR). This formulation avoids the generally discrete nature of segmentation problems and the inherent NP-hardness of the integer optimization associated. We extend the Segmentation via the Constraint Split Augmented Lagrangian Shrinkage (SegSALSA) algorithm by generalizing the vectori…
RWTH Aachen University（亚琛工业大学）；University of Texas at Austin（德克萨斯大学奥斯汀分校）
Hyperspectral images provide a rich representation of the underlying spectrum for each pixel, allowing for a pixel-wise classification/segmentation into different classes. As the acquisition of labeled training data is very time-consuming, unsupervised methods become crucial in hyperspectral image analysis. The spectral variability and noise in hyperspectral data make this task very challenging and define special requirements for such methods. Here, we present a novel unsupervised hyperspectral segmentation framework. It starts with a denoising and dimensionality reduction step by the well-est…
Istituto Nazionale di Fisica Nucleare（意大利国家核物理研究所）；Università Milano−Bicocca（米兰比可卡大学）
In this paper, we propose an unsupervised method for hyperspectral remote sensing image segmentation. The method exploits the mean-shift clustering algorithm that takes as input a preliminary hyperspectral superpixels segmentation together with the spectral pixel information. The proposed method does not require the number of segmentation classes as input parameter, and it does not exploit any a-priori knowledge about the type of land-cover or land-use to be segmented (e.g. water, vegetation, building etc.). Experiments on Salinas, SalinasA, Pavia Center and Pavia University datasets are carri…
Centre de Morphologie Mathématique（数学形态学中心）；Ecole des Mines de Paris（巴黎高等矿业学院）
The present paper develops a general methodology for the morphological segmentation of hyperspectral images, i.e., with an important number of channels. This approach, based on watershed, is composed of a spectral classification to obtain the markers and a vectorial gradient which gives the spatial information. Several alternative gradients are adapted to the different hyperspectral functions. Data reduction is performed either by Factor Analysis or by model fitting. Image segmentation is done on different spaces: factor space, parameters space, etc. On all these spaces the spatial/spectral se…
Various saliency detection algorithms from color images have been proposed to mimic eye fixation or attentive object detection response of human observers for the same scenes. However, developments on hyperspectral imaging systems enable us to obtain redundant spectral information of the observed scenes from the reflected light source from objects. A few studies using low-level features on hyperspectral images demonstrated that salient object detection can be achieved. In this work, we proposed a salient object detection model on hyperspectral images by applying manifold ranking (MR) on self-s…
Image segmentation is a fundamental step for the interpretation of Remote Sensing Images. Clustering or segmentation methods usually precede the classification task and are used as support tools for manual labeling. The most common algorithms, such as k-means, mean-shift, and MRS, require an extra manual step to find the scale parameter. The segmentation results are severely affected if the parameters are not correctly tuned and diverge from the optimal values. Additionally, the search for the optimal scale is a costly task, as it requires a comprehensive hyper-parameter search. This paper pro…
Hyperspectral image segmentation with a machine learning model trained using quantum annealer
AGH University（AGH大学）；Systems Research Institute Polish Academy of Sciences（波兰科学院系统研究所）；Nicolaus Copernicus Astronomical Center, Polish Academy of Sciences（波兰科学院尼古拉·哥白尼天文中心）；Center of Excellence in Artificial Intelligence, AGH University（人工智能卓越中心，AGH大学）
East China Normal University（华东师范大学）；Johns Hopkins University（约翰斯·霍普金斯大学）；Shanghai Jiaotong University（上海交通大学）
Hyperspectral imaging (HSI) unlocks the huge potential to a wide variety of applications relied on high-precision pathology image segmentation, such as computational pathology and precision medicine. Since hyperspectral pathology images benefit from the rich and detailed spectral information even beyond the visible spectrum, the key to achieve high-precision hyperspectral pathology image segmentation is to felicitously model the context along high-dimensional spectral bands. Inspired by the strong context modeling ability of transformers, we hereby, for the first time, formulate the contextual…
Hyperspectral imaging is an important sensing technology with broad applications and impact in areas including environmental science, weather, and geo/space exploration. One important task of hyperspectral image (HSI) processing is the extraction of spectral-spatial features. Leveraging on the recent-developed graph signal processing over multilayer networks (M-GSP), this work proposes several approaches to HSI segmentation based on M-GSP feature extraction. To capture joint spectral-spatial information, we first customize a tensor-based multilayer network (MLN) model for HSI, and define a MLN…
Nanjing University（南京大学）；Beijing Institute of Technology（北京理工大学）；University of Amsterdam（阿姆斯特丹大学）
High-resolution hyperspectral images (HSIs) contain the response of each pixel in different spectral bands, which can be used to effectively distinguish various objects in complex scenes. While HSI cameras have become low cost, algorithms based on it have not been well exploited. In this paper, we focus on a novel topic, weakly-supervised semantic segmentation in cityscape via HSIs. It is based on the idea that high-resolution HSIs in city scenes contain rich spectral information, which can be easily associated to semantics without manual labeling. Therefore, it enables low cost, highly reliab…
Addressing Annotation Scarcity in Hyperspectral Brain Image Segmentation with Unsupervised Domain Adaptation
Technical University of Munich, Munich, Germany（慕尼黑技术大学）；Department of Computing, Imperial College London, London, UK（伦敦帝国学院计算机系）；Munich Center for Machine Learning (MCML), Munich, Germany（慕尼黑机器学习中心（MCML））
Omni-Fusion of Spatial and Spectral for Hyperspectral Image Segmentation
Shanghai Key Laboratory of Multidimensional Information Processing, East China Normal University, Shanghai 200241, China（多维信息处理上海实验室，华东师范大学，上海200241，中国）
Autonomous navigation in unstructured off-road environments is greatly improved by semantic scene understanding. Conventional image processing algorithms are difficult to implement and lack robustness due to a lack of structure and high variability across off-road environments. The use of neural networks and machine learning can overcome the previous challenges but they require large labeled data sets for training. In our work we propose the use of hyperspectral images for real-time pixel-wise semantic classification and segmentation, without the need of any prior training data. The resulting …
Institute of Integrated and Intelligent Systems, Griffith University（格里菲斯大学集成与智能系统研究所）；School of Engineering and Information Technology, University of New South Wales at Canberra（新南威尔士大学堪培拉校区工程与信息技术学院）；Laboratoire Grenoblois de l’Image, de la Parole, du Signal et de l’Automatique (GIPSA-Lab), Grenoble Institute of Technology（格勒诺布尔理工学院图像、语音、信号与自动化实验室）；Faculty of Electrical and Computer Engineering, University of Iceland（冰岛大学电气与计算机工程学院）
Image segmentation is considered to be one of the critical tasks in hyperspectral remote sensing image processing. Recently, convolutional neural network (CNN) has established itself as a powerful model in segmentation and classification by demonstrating excellent performances. The use of a graphical model such as a conditional random field (CRF) contributes further in capturing contextual information and thus improving the segmentation performance. In this paper, we propose a method to segment hyperspectral images by considering both spectral and spatial information via a combined framework c…
Evaluation of Deep Learning Semantic Segmentation for Land Cover Mapping on Multispectral, Hyperspectral and High Spatial Aerial Imagery
Center for Space and Remote Sensing Research, National Central University（国立中央大学太空及遥测研究中心）；Department of Computer Science and Information Engineering, National Central University（国立中央大学资讯工程学系）
Visual discrimination of clinical tissue types remains challenging, with traditional RGB imaging providing limited contrast for such tasks. Hyperspectral imaging (HSI) is a promising technology providing rich spectral information that can extend far beyond three-channel RGB imaging. Moreover, recently developed snapshot HSI cameras enable real-time imaging with significant potential for clinical applications. Despite this, the investigation into the relative performance of HSI over RGB imaging for semantic segmentation purposes has been limited, particularly in the context of medical imaging. …
Herein, we present a system for hyperspectral image segmentation that utilizes multiple class--based denoising autoencoders which are efficiently trained. Moreover, we present a novel hyperspectral data augmentation method for labelled HSI data using linear mixtures of pixels from each class, which helps the system with edge pixels which are almost always mixed pixels. Finally, we utilize a deep neural network and morphological hole-filling to provide robust image classification. Results run on the Salinas dataset verify the high performance of the proposed algorithm.
German Cancer Research Center (DKFZ)（德国癌症研究中心）；Helmholtz Information and Data Science School for Health（亥姆霍兹健康信息与数据科学学院）；Heidelberg University Hospital（海德堡大学医院）；Heidelberg University（海德堡大学）
Semantic image segmentation is an important prerequisite for context-awareness and autonomous robotics in surgery. The state of the art has focused on conventional RGB video data acquired during minimally invasive surgery, but full-scene semantic segmentation based on spectral imaging data and obtained during open surgery has received almost no attention to date. To address this gap in the literature, we are investigating the following research questions based on hyperspectral imaging (HSI) data of pigs acquired in an open surgery setting: (1) What is an adequate representation of HSI data for…
Semantic segmentation of surgical hyperspectral images under geometric domain shifts
German Cancer Research Center (DKFZ)（德国癌症研究中心）；Heidelberg University（海德堡大学）；National Center for Tumor Diseases (NCT)（国家肿瘤疾病中心）；Heidelberg University Hospital（海德堡大学医院）
Handling Geometric Domain Shifts in Semantic Segmentation of Surgical RGB and Hyperspectral Images
German Cancer Research Center (DKFZ)（德国癌症研究中心）；Helmholtz Information and Data Science School for Health（亥姆霍兹健康信息与数据科学学院）；Heidelberg University（海德堡大学）；National Center for Tumor Diseases (NCT)（国家肿瘤疾病中心）；Heidelberg University Hospital（海德堡大学医院）；Medical Faculty, Heidelberg University（海德堡大学医学院）
AMBER -- Advanced SegFormer for Multi-Band Image Segmentation: an application to Hyperspectral Imaging
University of Naples Federico II（那不勒斯费德里科二世大学）；INFN – Section of Naples（国家核物理研究所那不勒斯分部）；Stazione Zoologica Anton Dohrn（安托万·多尔恩动物站）；INAF - Osservatorio Astronomico di Capodimonte（意大利国家天文研究所卡波迪蒙特天文台）
King Abdullah University of Science and Technology (KAUST)（阿卜杜拉国王科技大学）
Hyperspectral imaging has attracted significant attention to identify spectral signatures for image classification and automated pattern recognition in computer vision. State-of-the-art implementations of snapshot hyperspectral imaging rely on bulky, non-integrated, and expensive optical elements, including lenses, spectrometers, and filters. These macroscopic components do not allow fast data processing for, e.g real-time and high-resolution videos. This work introduces Hyplex, a new integrated architecture addressing the limitations discussed above. Hyplex is a CMOS-compatible, fast hyperspe…
Nowadays, hyperspectral image classification widely copes with spatial information to improve accuracy. One of the most popular way to integrate such information is to extract hierarchical features from a multiscale segmentation. In the classification context, the extracted features are commonly concatenated into a long vector (also called stacked vector), on which is applied a conventional vector-based machine learning technique (e.g. SVM with Gaussian kernel). In this paper, we rather propose to use a sequence structured kernel: the spectrum kernel. We show that the conventional stacked vect…
College of Software, Nankai University（南开大学软件学院）；School of Information Engineering, Zhengzhou University（郑州大学信息工程学院）
The Hyperspectral image (HSI) classification is a standard remote sensing task, in which each image pixel is given a label indicating the physical land-cover on the earth's surface. The achievements of image semantic segmentation and deep learning approaches on ordinary images have accelerated the research on hyperspectral image classification. Moreover, the utilization of both the spectral and spatial cues in hyperspectral images has shown improved classification accuracy in hyperspectral image classification. The use of only 3D Convolutional Neural Networks (3D-CNN) to extract both spatial a…
Pontifical Catholic University of Rio de Janeiro（里约热内卢天主教大学）；IBM Research（IBM研究院）；McMaster University（麦克马斯特大学）；National Institute for Space Research (INPE)（巴西国家空间研究所）；Santa Catarina State University (UDESC)（圣卡塔琳娜州立大学）
This work proposes a multi-task fully convolutional architecture for tree species mapping in dense forests from sparse and scarce polygon-level annotations using hyperspectral UAV-borne data. Our model implements a partial loss function that enables dense tree semantic labeling outcomes from non-dense training samples, and a distance regression complementary task that enforces tree crown boundary constraints and substantially improves the model performance. Our multi-task architecture uses a shared backbone network that learns common representations for both tasks and two task-specific decoder…
Tinto: Multisensor Benchmark for 3D Hyperspectral Point Cloud Segmentation in the Geosciences
Helmholtz-Zentrum Dresden-Rossendorf（德累斯顿-罗森多夫亥姆霍兹中心）；Helmholtz Institute Freiberg for Resource Technology（弗赖贝格亥姆霍兹资源技术研究所）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Institute of Advanced Research in Artificial Intelligence（高级人工智能研究所）
Tufts University（塔夫茨大学）；University of Cambridge（剑桥大学）；City University of Hong Kong（香港城市大学）；Wake Forest University（维克森林大学）
Ash dieback (Hymenoscyphus fraxineus) is an introduced fungal disease that is causing the widespread death of ash trees across Europe. Remote sensing hyperspectral images encode rich structure that has been exploited for the detection of dieback disease in ash trees using supervised machine learning techniques. However, to understand the state of forest health at landscape-scale, accurate unsupervised approaches are needed. This article investigates the use of the unsupervised Diffusion and VCA-Assisted Image Segmentation (D-VIS) clustering algorithm for the detection of ash dieback disease in…
Amazon Web Service（亚马逊云服务）；University of Houston（休斯顿大学）
Deep neural networks have proven to be very effective for computer vision tasks, such as image classification, object detection, and semantic segmentation -- these are primarily applied to color imagery and video. In recent years, there has been an emergence of deep learning algorithms being applied to hyperspectral and multispectral imagery for remote sensing and biomedicine tasks. These multi-channel images come with their own unique set of challenges that must be addressed for effective image analysis. Challenges include limited ground truth (annotation is expensive and extensive labeling i…
MWIR-4-Plastic: The Identification of Complex End-of-Life Industrial Plastic using Mid-wave Infrared Hyperspectral Imaging and Machine Learning
Helmholtz-Zentrum Dresden-Rossendorf (HZDR)（德累斯顿-罗森多夫亥姆霍兹中心）；Helmholtz Institute Freiberg for Resource Technology (HIF)（弗莱贝格亥姆霍兹资源技术研究所）；University of Antwerp（安特卫普大学）；Freie Universität Berlin（柏林自由大学）；University of Iceland（冰岛大学）；TU Bergakademie Freiberg（弗莱贝格工业大学）
Recent advances in computer vision using deep learning with RGB imagery (e.g., object recognition and detection) have been made possible thanks to the development of large annotated RGB image datasets. In contrast, multispectral image (MSI) and hyperspectral image (HSI) datasets contain far fewer labeled images, in part due to the wide variety of sensors used. These annotations are especially limited for semantic segmentation, or pixel-wise classification, of remote sensing imagery because it is labor intensive to generate image annotations. Low-shot learning algorithms can make effective infe…
Rochester Institute of Technology（罗切斯特理工学院）；ON Semiconductor（安森美半导体）
We investigate applying convolutional neural network (CNN) architecture to facilitate aerial hyperspectral scene understanding and present a new hyperspectral dataset-AeroRIT-that is large enough for CNN training. To date the majority of hyperspectral airborne have been confined to various sub-categories of vegetation and roads and this scene introduces two new categories: buildings and cars. To the best of our knowledge, this is the first comprehensive large-scale hyperspectral scene with nearly seven million pixel annotations for identifying cars, roads, and buildings. We compare the perform…
Multi-Modal Semantic Segmentation of Electrolyzer Components for Sustainable Hydrogen Technologies: A Dual-Branch Deep Learning Approach
Applied Artificial Intelligence and INtelligent Systems (AAIINS) Laboratory（应用人工智能与智能系统（AAIINS）实验室）；University of Scholars（学者大学）；Murdoch University（莫道克大学）；Charles Darwin University（查尔斯达尔文大学）
University of Missouri（密苏里大学）；University of Florida（佛罗里达大学）
A map-guided superpixel segmentation method for hyperspectral imagery is developed and introduced. The proposed approach develops a hyperspectral-appropriate version of the SLIC superpixel segmentation algorithm, leverages map information to guide segmentation, and incorporates the semi-supervised Partial Membership Latent Dirichlet Allocation (sPM-LDA) to obtain a final superpixel segmentation. The proposed method is applied to two real hyperspectral data sets and quantitative cluster validity metrics indicate that the proposed approach outperforms existing hyperspectral superpixel segmentati…
Carlson Center for Imaging Science, Rochester Institute of Technology（卡尔森成像科学中心，罗切斯特理工学院）
Deep learning continues to push state-of-the-art performance for the semantic segmentation of color (i.e., RGB) imagery; however, the lack of annotated data for many remote sensing sensors (i.e. hyperspectral imagery (HSI)) prevents researchers from taking advantage of this recent success. Since generating sensor specific datasets is time intensive and cost prohibitive, remote sensing researchers have embraced deep unsupervised feature extraction. Although these methods have pushed state-of-the-art performance on current HSI benchmarks, many of these tools are not readily accessible to many re…
Few-shot learning is a standard practice in most deep learning based histopathology image segmentation, given the relatively low number of digitized slides that are generally available. While many models have been developed for domain specific histopathology image segmentation, cross-domain generalization remains a key challenge for properly validating models. Here, tooling and datasets to benchmark model performance across histopathological domains are lacking. To address this limitation, we introduce MetaHistoSeg - a Python framework that implements unique scenarios in both meta learning and…
HyperVision: A Channel-Adaptive Ground-Based Hyperspectral Vision Pre-trained Backbone
Griffith University, Australia（格里菲斯大学，澳大利亚）；Wuhan University, China（武汉大学，中国）；Nanjing University of Science and Technology, China（南京理工大学，中国）；Huaiyin Normal University, China（淮阴师范学院，中国）；Massey University, New Zealand（马斯sey大学，新西兰）
In this paper, we propose an unified hyperspectral image classification method which takes three-dimensional hyperspectral data cube as an input and produces a classification map. In the proposed method, a deep neural network which uses spectral and spatial information together with residual connections, and pixel affinity network based segmentation-aware superpixels are used together. In the architecture, segmentation-aware superpixels run on the initial classification map of deep residual network, and apply majority voting on obtained results. Experimental results show that our propoped meth…
