With the convenient availability of remote sensing data, how to make models to interpret complex remote sensing data attracts wide attention. In remote sensing data, hyperspectral images contain spectral information and LiDAR contains elevation information. Hence, more explorations are warranted to better fuse the features of different source data. In this paper, we introduce semantic understanding to dynamically fuse data from two different sources, extract features of HSI and LiDAR through different capsule network branches and improve self-supervised loss and random rigid rotation in Canonical Capsule to a high-dimensional situation. Canonical Capsule computes the capsule decomposition of objects by permutation-equivariant attention and the process is self-supervised by training pairs of randomly rotated objects. After fusing the features of HSI and LiDAR with semantic understanding, the unsupervised extraction of spectral-spatial-elevation fusion features is achieved. With two real-world examples of HSI and LiDAR fused, the experimental results show that the proposed multi-branch high-dimensional canonical capsule algorithm can be effective for semantic understanding of HSI and LiDAR. It indicates that the model can extract HSI and LiDAR data features effectively as opposed to existing models for unsupervised extraction of multi-source RS data.
Sun Yat-sen University（中山大学）；Southern Marine Science and Engineering Guangdong Laboratory (Zhuhai)（南方海洋科学与工程广东省实验室（珠海））；Shenzhen University（深圳大学）
Previous studies have shown the great potential of capsule networks for the spatial contextual feature extraction from {hyperspectral images (HSIs)}. However, the sampling locations of the convolutional kernels of capsules are fixed and cannot be adaptively changed according to the inconsistent semantic information of HSIs. Based on this observation, this paper proposes an adaptive spatial pattern capsule network (ASPCNet) architecture by developing an adaptive spatial pattern (ASP) unit, that can rotate the sampling location of convolutional kernels on the basis of an enlarged receptive field…
Liaoning Technical University（辽宁工程技术大学）；Fujian Institute of Research on the Structure of Matter, Chinese Academy of Sciences（中国科学院福建物质结构研究所）；East China Normal University（华东师范大学）；The Hong Kong Polytechnic University（香港理工大学）
Recently, convolutional neural networks (CNNs) have achieved excellent performances in many computer vision tasks. Specifically, for hyperspectral images (HSIs) classification, CNNs often require very complex structure due to the high dimension of HSIs. The complex structure of CNNs results in prohibitive training efforts. Moreover, the common situation in HSIs classification task is the lack of labeled samples, which results in accuracy deterioration of CNNs. In this work, we develop an easy-to-implement capsule network to alleviate the aforementioned problems, i.e., 1D-convolution capsule ne…
Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education of China（教育部智能感知与图像理解重点实验室）；School of Computer Science and Technology（计算机科学与技术学院）；Xidian University（西安电子科技大学）；School of Artificial Intelligence（人工智能学院）；Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education（教育部智能感知与图像理解重点实验室）；School of Electronic Engineering（电子工程学院）
Remote Sensing Technology Institute (IMF), German Aerospace Center (DLR)（德国航空航天中心遥感技术研究所）；Signal Processing in Earth Observation (SiPEO), Technische Universität München (TUM)（慕尼黑工业大学地球观测信号处理中心）；Faculty of Electrical and Computer Engineering, University of Iceland（冰岛大学电气与计算机工程学院）；Department of Technology of Computers and Communications, University of Extremadura（埃斯特雷马杜拉大学计算机与通信技术系）
In this paper, an approach is proposed to fuse LiDAR and hyperspectral data, which considers both spectral and spatial information in a single framework. Here, an extended self-dual attribute profile (ESDAP) is investigated to extract spatial information from a hyperspectral data set. To extract spectral information, a few well-known classifiers have been used such as support vector machines (SVMs), random forests (RFs), and artificial neural networks (ANNs). The proposed method accurately classify the relatively volumetric data set in a few CPU processing time in a real ill-posed situation wh…
An efficient linear self-attention fusion model is proposed in this paper for the task of hyperspectral image (HSI) and LiDAR data joint classification. The proposed method is comprised of a feature extraction module, an attention module, and a fusion module. The attention module is a plug-and-play linear self-attention module that can be extensively used in any model. The proposed model has achieved the overall accuracy of 95.40\% on the Houston dataset. The experimental results demonstrate the superiority of the proposed method over other state-of-the-art models.
Toward Interpretable Multimodal Fusion: Heat Conduction Modeling for Hyperspectral and LiDAR Joint Classification
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；Key Laboratory of Target Cognition and Application Technology（目标认知与应用技术重点实验室）；School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences（中国科学院大学电子电气与通信工程学院）；State Key Laboratory of Remote Sensing and Digital Earth, Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院遥感与数字地球国家重点实验室）；Helmholtz-Zentrum Dresden-Rossendorf（亥姆霍兹德累斯顿罗森多夫研究中心）；Faculty of Electrical and Computer Engineering, University of Iceland（冰岛大学电气与计算机工程系）
LiDAR-Guided Cross-Attention Fusion for Hyperspectral Band Selection and Image Classification
School of Information and Communication Technology, Griffith University（格里菲斯大学信息与通信技术学院）；Queensland Department of Agriculture and Fisheries（昆士兰州农业与渔业厅）
University of Cambridge（剑桥大学）；Fondazione E. Mach（埃马努埃莱·马赫基金会）；University of Oxford（牛津大学）；The University of Queensland（昆士兰大学）；Natural England（英国自然署）
Recognising individual trees within remotely sensed imagery has important applications in forest ecology and management. Several algorithms for tree delineation have been suggested, mostly based on locating local maxima or inverted basins in raster canopy height models (CHMs) derived from Light Detection And Ranging (LiDAR) data or photographs. However, these algorithms often lead to inaccurate estimates of forest stand characteristics due to the limited information content of raster CHMs. Here we develop a 3D tree delineation method which uses graph cut to delineate trees from the full 3D LiD…
German Aerospace Center (DLR)（德国航空航天中心）；Technical University of Munich (TUM)（慕尼黑工业大学）；Univ. Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；Inria（法国国家信息与自动化研究所）；CNRS（法国国家科学研究中心）；Grenoble INP（格勒诺布尔国立理工学院）；University of Iceland（冰岛大学）；RIKEN（日本理化学研究所）
Due to the ever-growing diversity of the data source, multi-modality feature learning has attracted more and more attention. However, most of these methods are designed by jointly learning feature representation from multi-modalities that exist in both training and test sets, yet they are less investigated in absence of certain modality in the test phase. To this end, in this letter, we propose to learn a shared feature space across multi-modalities in the training process. By this way, the out-of-sample from any of multi-modalities can be directly projected onto the learned space for a more e…
University of Padova（帕多瓦大学）；Fondazione Edmund Mach（埃德蒙·马赫基金会）；Provincia Autonoma di Trento（特伦托自治省）
This work intends to lay the foundations for identifying the prevailing forest types and the delineation of forest units within private forest inventories in the Autonomous Province of Trento (PAT), using currently available remote sensing solutions. In particular, data from LiDAR and hyperspectral surveys of 2014 made available by PAT were acquired and processed. Such studies are very important in the context of forest management scenarios. The method includes defining tree species ground-truth by outlining single tree crowns with polygons and labeling them. Successively two supervised machin…
Dynamic Cross-Modal Feature Interaction Network for Hyperspectral and LiDAR Data Classification
School of Computer Science and Technology, Ocean University of China（中国海洋大学计算机科学与技术学院）；Mississippi State University（密西西比州立大学）；Chongqing University of Posts and Telecommunications（重庆邮电大学）
Nanjing University of Information Science and Technology（南京信息工程大学）；University of Missouri-Kansas City（密苏里大学堪萨斯城分校）；Helmholtz-Zentrum Dresden-Rossendorf（德累斯顿-罗森多夫亥姆霍兹中心）；Helmholtz Institute Freiberg for Resource Technology（弗赖贝格资源技术亥姆霍兹研究所）；German Aerospace Center（德国航空航天中心）；Technical University of Munich（慕尼黑工业大学）
In this paper, we propose an efficient and effective framework to fuse hyperspectral and Light Detection And Ranging (LiDAR) data using two coupled convolutional neural networks (CNNs). One CNN is designed to learn spectral-spatial features from hyperspectral data, and the other one is used to capture the elevation information from LiDAR data. Both of them consist of three convolutional layers, and the last two convolutional layers are coupled together via a parameter sharing strategy. In the fusion phase, feature-level and decision-level fusion methods are simultaneously used to integrate the…
Ocean University of China（中国海洋大学）；Southwest Jiaotong University（西南交通大学）；Mississippi State University（密西西比州立大学）
The joint hyperspectral image (HSI) and LiDAR data classification aims to interpret ground objects at more detailed and precise level. Although deep learning methods have shown remarkable success in the multisource data classification task, self-supervised learning has rarely been explored. It is commonly nontrivial to build a robust self-supervised learning model for multisource data classification, due to the fact that the semantic similarities of neighborhood regions are not exploited in existing contrastive learning framework. Furthermore, the heterogeneous gap induced by the inconsistent …
There is much current interest in using multi-sensor airborne remote sensing to monitor the structure and biodiversity of forests. This paper addresses the application of non-parametric image registration techniques to precisely align images obtained from multimodal imaging, which is critical for the successful identification of individual trees using object recognition approaches. Non-parametric image registration, in particular the technique of optimizing one objective function containing data fidelity and regularization terms, provides flexible algorithms for image registration. Using a sur…
Manchester Metropolitan University（曼彻斯特城市大学）；Hebei Agriculture University（河北农业大学）
Late blight disease is one of the most destructive diseases in potato crop, leading to serious yield losses globally. Accurate diagnosis of the disease at early stage is critical for precision disease control and management. Current farm practices in crop disease diagnosis are based on manual visual inspection, which is costly, time consuming, subject to individual bias. Recent advances in imaging sensors (e.g. RGB, multiple spectral and hyperspectral cameras), remote sensing and machine learning offer the opportunity to address this challenge. Particularly, hyperspectral imagery (HSI) combini…
University of Missouri（密苏里大学）；University of Michigan（密歇根大学）；University of Florida（佛罗里达大学）
In remote sensing, each sensor can provide complementary or reinforcing information. It is valuable to fuse outputs from multiple sensors to boost overall performance. Previous supervised fusion methods often require accurate labels for each pixel in the training data. However, in many remote sensing applications, pixel-level labels are difficult or infeasible to obtain. In addition, outputs from multiple sensors often have different resolution or modalities. For example, rasterized hyperspectral imagery presents data in a pixel grid while airborne Light Detection and Ranging (LiDAR) generates…
Northwestern Polytechnical University（西北工业大学）；Université Sorbonne Paris Nord（索邦巴黎北大学）；Xidian University（西安电子科技大学）
To benefit the complementary information between heterogeneous data, we introduce a new Multimodal Transformer (MMFormer) for Remote Sensing (RS) image classification using Hyperspectral Image (HSI) accompanied by another source of data such as Light Detection and Ranging (LiDAR). Compared with traditional Vision Transformer (ViT) lacking inductive biases of convolutions, we first introduce convolutional layers to our MMFormer to tokenize patches from multimodal data of HSI and LiDAR. Then we propose a Multi-scale Multi-head Self-Attention (MSMHSA) module to address the problem of compatibilit…
Manchester Metropolitan University（曼彻斯特城市大学）；Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；Brunel University（布鲁内尔大学）
Spectral-spatial based deep learning models have recently proven to be effective in hyperspectral image (HSI) classification for various earth monitoring applications such as land cover classification and agricultural monitoring. However, due to the nature of "black-box" model representation, how to explain and interpret the learning process and the model decision, especially for vegetation classification, remains an open challenge. This study proposes a novel interpretable deep learning model -- a biologically interpretable two-stage deep neural network (BIT-DNN), by incorporating the prior-k…
Centre of Studies in Resources Engineering, Indian Institute of Technology Bombay（印度理工学院孟买分校资源工程研究中心）
As the field of remote sensing is evolving, we witness the accumulation of information from several modalities, such as multispectral (MS), hyperspectral (HSI), LiDAR etc. Each of these modalities possess its own distinct characteristics and when combined synergistically, perform very well in the recognition and classification tasks. However, fusing multiple modalities in remote sensing is cumbersome due to highly disparate domains. Furthermore, the existing methods do not facilitate cross-modal interactions. To this end, we propose a novel transformer based fusion method for HSI and LiDAR mod…
SLIMBRAIN: Augmented Reality Real-Time Acquisition and Processing System For Hyperspectral Classification Mapping with Depth Information for In-Vivo Surgical Procedures
Research Center on Software Technologies and Multimedia Systems, Universidad Politécnica de Madrid（马德里理工大学软件技术与多媒体系统研究中心）；Department of Neurosurgery, Hospital Universitario 12 de Octubre, Universidad Complutense de Madrid（马德里康普顿斯大学十月十二日大学医院神经外科）
The problem of effectively exploiting the information multiple data sources has become a relevant but challenging research topic in remote sensing. In this paper, we propose a new approach to exploit the complementarity of two data sources: hyperspectral images (HSIs) and light detection and ranging (LiDAR) data. Specifically, we develop a new dual-channel spatial, spectral and multiscale attention convolutional long short-term memory neural network (called dual-channel A3CLNN) for feature extraction and classification of multisource remote sensing data. Spatial, spectral and multiscale attent…
Southwest Jiaotong University（西南交通大学）；Xinyang Normal University（信阳师范大学）
Over the past few decades, with the rapid development of global aerospace and aerial remote sensing technology, the types of sensors have evolved from the traditional monomodal sensors (e.g., optical sensors) to the new generation of multimodal sensors [e.g., multispectral, hyperspectral, light detection and ranging (LiDAR) and synthetic aperture radar (SAR) sensors]. These advanced devices can dynamically provide various and abundant multimodal remote sensing images with different spatial, temporal, and spectral resolutions according to different application requirements. Since then, it is of…
With the emergence of passive and active optical sensors available for geospatial imaging, information fusion across sensors is becoming ever more important. An important aspect of single (or multiple) sensor geospatial image analysis is feature extraction - the process of finding "optimal" lower dimensional subspaces that adequately characterize class-specific information for subsequent analysis tasks, such as classification, change and anomaly detection etc. In recent work, we proposed and developed an angle-based discriminant analysis approach that projected data onto subspaces with maximal…
Dynamic Frequency Feature Fusion Network for Multi-Source Remote Sensing Data Classification
State Key Laboratory of Physical Oceanography, Ocean University of China（中国海洋大学物理海洋学国家重点实验室）；Department of Electrical and Computer Engineering, Mississippi State University（密苏里州立大学电气与计算机工程系）
Representative Spectral Correlation Network for Multi-source Remote Sensing Image Classification
State Key Laboratory of Physical Oceanography, Ocean University of China（中国海洋大学物理海洋学国家重点实验室）；Department of Computing, The Hong Kong Polytechnic University（香港理工大学计算机系）；Department of Electrical and Computer Engineering, Mississippi State University（密苏里州立大学电气与计算机工程系）
Department of Forest Sciences, University of Helsinki（赫尔辛基大学森林科学系）；School of Computer Science, National Engineering Research Center for Multimedia Software, Institute of Artificial Intelligence, and Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan University, Wuhan, China（武汉大学计算机学院、国家多媒体软件工程研究中心、人工智能研究院、湖北省多媒体与网络通信工程重点实验室，武汉，中国）
Multimodal Fusion Transformer for Remote Sensing Image Classification
Jalpaiguri Government Engineering College（贾尔派古里政府工程学院）；Technical University of Munich（慕尼黑工业大学）；Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；Helmholtz-Zentrum Dresden-Rossendorf（亥姆霍兹德累斯顿罗森多夫研究中心）；University of Extremadura（埃斯特雷马杜拉大学）；Univ. Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；CNRS（法国国家科学研究中心）；Grenoble INP（格勒诺布尔国立理工学院）
