遥感检索增强生成：通过多模态数据集和检索增强生成模型连接遥感图像与综合知识
Remote Sensing Retrieval-Augmented Generation: Bridging Remote Sensing Imagery and Comprehensive Knowledge with a Multi-Modal Dataset and Retrieval-Augmented Generation Model
AI总结：
本文提出RS-RAG框架，通过多模态数据集和检索增强生成模型，提升遥感图像与综合知识的连接能力，有效提升复杂查询的语义推理性能。
摘要：
近年来，视觉语言模型（VLMs）在自然图像领域展现出了出色的跨任务能力。受这些进展的启发，遥感社区开始采用VLMs进行遥感视觉语言任务，包括场景理解、图像描述和视觉问答。然而，现有的遥感VLMs通常依赖封闭集场景理解和专注于通用场景描述，但缺乏整合外部知识的能力。这一限制阻碍了它们在涉及领域特定或世界知识的复杂或上下文依赖查询中的语义推理能力。为了解决这些挑战，我们首先引入了一个多模态的遥感世界知识（RSWK）数据集，该数据集包含14141个知名地标（来自175个国家）的高分辨率卫星图像和详细的文本描述，整合了遥感领域知识和更广泛的世界知识。基于此数据集，我们提出了一种新的遥感检索增强生成（RS-RAG）框架，包含两个关键组件。多模态知识向量数据库构建模块将遥感图像和相关文本知识编码到统一的向量空间中。知识检索与响应生成模块根据图像和/或文本查询检索并重新排序相关知识，并将检索内容整合到知识增强的提示中，以指导VLM生成语境相关的响应。我们在三个代表性的视觉语言任务上验证了我们的方法的有效性，包括图像描述、图像分类和视觉问答，在这些任务中，RS-RAG显著优于最先进的基线模型。
英文摘要：
Recent progress in VLMs has demonstrated impressive capabilities across a variety of tasks in the natural image domain. Motivated by these advancements, the remote sensing community has begun to adopt VLMs for remote sensing vision-language tasks, including scene understanding, image captioning, and visual question answering. However, existing remote sensing VLMs typically rely on closed-set scene understanding and focus on generic scene descriptions, yet lack the ability to incorporate external knowledge. This limitation hinders their capacity for semantic reasoning over complex or context-dependent queries that involve domain-specific or world knowledge. To address these challenges, we first introduced a multimodal Remote Sensing World Knowledge (RSWK) dataset, which comprises high-resolution satellite imagery and detailed textual descriptions for 14,141 well-known landmarks from 175 countries, integrating both remote sensing domain knowledge and broader world knowledge. Building upon this dataset, we proposed a novel Remote Sensing Retrieval-Augmented Generation (RS-RAG) framework, which consists of two key components. The Multi-Modal Knowledge Vector Database Construction module encodes remote sensing imagery and associated textual knowledge into a unified vector space. The Knowledge Retrieval and Response Generation module retrieves and re-ranks relevant knowledge based on image and/or text queries, and incorporates the retrieved content into a knowledge-augmented prompt to guide the VLM in producing contextually grounded responses. We validated the effectiveness of our approach on three representative vision-language tasks, including image captioning, image classification, and visual question answering, where RS-RAG significantly outperformed state-of-the-art baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
