使用 VLM 和双交叉注意力网络的多模态遥感场景分类
Multimodal Remote Sensing Scene Classification Using VLMs and Dual-Cross Attention Networks
AI总结：
针对遥感场景分类中图像单模态局限及文本标注昂贵问题，提出利用 VLM 生成文本并通过双交叉注意力融合的多模态框架，在五个数据集上表现优于基线，并支持零样本未见类分类。
摘要：
遥感场景分类（RSSC）是一项关键任务，在土地利用和资源管理中具有多种应用。尽管基于单模态图像的方法展现出良好前景，但它们往往受到类内差异大、类间相似度高等限制。引入文本信息可通过提供额外上下文和语义理解来增强分类，然而人工文本标注既费力又昂贵。本文提出一种新颖的 RSSC 框架，将大型视觉-语言模型（VLM）生成的文本描述作为辅助模态加以整合，且不会产生昂贵的人工标注成本。为充分利用视觉与文本数据之间潜在的互补性，我们提出一种基于双交叉注意力的网络，将这些模态融合为统一表示。在五个 RSSC 数据集上进行的大量定量与定性评估实验表明，我们的框架始终优于基线模型。我们还验证了 VLM 生成的文本描述相比人工标注描述的有效性。此外，我们设计了零样本分类场景，以证明所学多模态表示能够有效用于未见类别分类。本研究为在 RSSC 任务中利用文本信息开辟了新机遇，并提供了一种有前景的多模态融合结构，为未来研究提供见解与启发。代码可在 https://github.com/CJR7/MultiAtt-RSSC 获取。
英文摘要：
Remote sensing scene classification (RSSC) is a critical task with diverse applications in land use and resource management. While unimodal image-based approaches show promise, they often struggle with limitations such as high intra-class variance and inter-class similarity. Incorporating textual information can enhance classification by providing additional context and semantic understanding, but manual text annotation is labor-intensive and costly. In this work, we propose a novel RSSC framework that integrates text descriptions generated by large vision-language models (VLMs) as an auxiliary modality without incurring expensive manual annotation costs. To fully leverage the latent complementarities between visual and textual data, we propose a dual cross-attention-based network to fuse these modalities into a unified representation. Extensive experiments with both quantitative and qualitative evaluation across five RSSC datasets demonstrate that our framework consistently outperforms baseline models. We also verify the effectiveness of VLM-generated text descriptions compared to human-annotated descriptions. Additionally, we design a zero-shot classification scenario to show that the learned multimodal representation can be effectively utilized for unseen class classification. This research opens new opportunities for leveraging textual information in RSSC tasks and provides a promising multimodal fusion structure, offering insights and inspiration for future studies. Code is available at: https://github.com/CJR7/MultiAtt-RSSC
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
