通过基于变换器的异质显著图表示提升多模态遥感图像分类
Boosting Multimodal Remote Sensing Image Classification with Transformer-based Heterogeneously Salient Graph Representation
AI总结：
本文提出基于变换器的异质显著图表示方法，解决多模态遥感图像分类中特征表示不充分、长距离依赖建模复杂及过拟合问题，通过异质图编码器、多卷积调节器和均值前向策略提升分类性能。
摘要：
不同模态的数据可以提供丰富的互补信息，例如高光谱图像（HSI）提供丰富的光谱-空间特性，合成孔径雷达（SAR）提供地球表面的结构信息，激光雷达（LiDAR）提供地面高程信息。因此，结合多模态图像进行精细准确的土地覆盖解释是一个自然的想法。尽管已经有许多努力尝试实现多源遥感图像分类，但仍存在三个问题：1）缺乏充分考虑模态异质性的特征表示；2）大量特征和复杂计算与建模长距离依赖相关；3）稀疏标注样本导致的过拟合现象。为克服上述障碍，本文提出了一种基于变换器的异质显著图表示（THSGR）方法。首先，提出一个多模态异质图编码器，以编码异构数据中不同的非欧几里得结构特征。然后，设计了一个无自注意力的多卷积调节器，以有效且高效地建模长期依赖性。最后，开发了一种均值前向策略以避免过拟合。基于上述结构，所提出的模型能够突破模态差距，获得具有竞争力的时间成本的差异化图表示，即使在少量训练样本的情况下。在三个基准数据集上与各种最先进的（SOTA）方法的实验和分析显示了所提THSGR的性能。代码将在https://github.com/jqyang22上提供。
英文摘要：
Data collected by different modalities can provide a wealth of complementary information, such as hyperspectral image (HSI) to offer rich spectral-spatial properties, synthetic aperture radar (SAR) to provide structural information about the Earth's surface, and light detection and ranging (LiDAR) to cover altitude information about ground elevation. Therefore, a natural idea is to combine multimodal images for refined and accurate land-cover interpretation. Although many efforts have been attempted to achieve multi-source remote sensing image classification, there are still three issues as follows: 1) indiscriminate feature representation without sufficiently considering modal heterogeneity, 2) abundant features and complex computations associated with modeling long-range dependencies, and 3) overfitting phenomenon caused by sparsely labeled samples. To overcome the above barriers, a transformer-based heterogeneously salient graph representation (THSGR) approach is proposed in this paper. First, a multimodal heterogeneous graph encoder is presented to encode distinctively non-Euclidean structural features from heterogeneous data. Then, a self-attention-free multi-convolutional modulator is designed for effective and efficient long-term dependency modeling. Finally, a mean forward strategy is developed in order to avoid overfitting. Based on the above structures, the proposed model is able to break through modal gaps to obtain differentiated graph representation with competitive time cost, even for a small fraction of training samples. Experiments and analyses in three benchmark datasets with various state-of-the-art (SOTA) approaches show the performance of the proposed THSGR. The code will be available in https://github.com/jqyang22.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
