面向遥感图像描述的自举式交互图像-文本对齐方法
Bootstrapping Interactive Image-Text Alignment for Remote Sensing Image Captioning
AI总结：
本文提出BITA方法，通过两阶段视觉-语言预训练和轻量级交互式Fourier Transformer实现遥感图像与文本的高效对齐，在多个数据集上取得优于先进方法的图像描述性能。
摘要：
近年来，遥感图像描述在遥感领域引起了广泛关注。由于遥感图像在空间分辨率上存在显著差异，该领域的现有方法主要集中于遥感图像特征的细粒度提取，但无法有效处理视觉特征与文本特征之间的语义一致性。为了高效地对齐图像与文本，我们提出了一种新颖的基于两阶段视觉-语言预训练的方法，用于自举式交互图像-文本对齐以实现遥感图像描述，称为BITA。该方法依赖于一个轻量级交互式Fourier Transformer的设计，以更好地对齐遥感图像-文本特征。交互式Fourier Transformer中的Fourier层能够在频域中提取遥感图像的多尺度特征，从而减少遥感视觉特征的冗余。具体而言，第一阶段通过图像-文本对比学习进行初步对齐，将从交互式Fourier Transformer中学习到的多尺度遥感特征与文本特征对齐。在第二阶段，交互式Fourier Transformer将冻结的图像编码器与一个大语言模型连接起来。然后，利用前缀因果语言建模来引导基于视觉特征的文本生成过程。最终，在UCM-caption、RSICD和NWPU-caption数据集上，实验结果清楚地表明BITA优于其他先进的对比方法。代码可在https://github.com/yangcong356/BITA获取。
英文摘要：
Recently, remote sensing image captioning has gained significant attention in the remote sensing community. Due to the significant differences in spatial resolution of remote sensing images, existing methods in this field have predominantly concentrated on the fine-grained extraction of remote sensing image features, but they cannot effectively handle the semantic consistency between visual features and textual features. To efficiently align the image-text, we propose a novel two-stage vision-language pre-training-based approach to bootstrap interactive image-text alignment for remote sensing image captioning, called BITA, which relies on the design of a lightweight interactive Fourier Transformer to better align remote sensing image-text features. The Fourier layer in the interactive Fourier Transformer is capable of extracting multi-scale features of remote sensing images in the frequency domain, thereby reducing the redundancy of remote sensing visual features. Specifically, the first stage involves preliminary alignment through image-text contrastive learning, which aligns the learned multi-scale remote sensing features from the interactive Fourier Transformer with textual features. In the second stage, the interactive Fourier Transformer connects the frozen image encoder with a large language model. Then, prefix causal language modeling is utilized to guide the text generation process using visual features. Ultimately, across the UCM-caption, RSICD, and NWPU-caption datasets, the experimental results clearly demonstrate that BITA outperforms other advanced comparative approaches. The code is available at https://github.com/yangcong356/BITA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
