RS-GPT4V：面向遥感图像理解的统一多模态指令跟随数据集
RS-GPT4V: A Unified Multimodal Instruction-Following Dataset for Remote Sensing Image Understanding
AI总结：
针对遥感图像理解新范式下旧数据集不适配的问题，本文构建了具备泛化性、复杂场景理解、推理能力的统一多模态指令跟随数据集RS-GPT4V，实验验证其可提升MLLM的细粒度描述能力。
摘要：
在多模态大语言模型（MLLM）的推动下，遥感图像智能理解模型正经历一场全新的深刻范式转变：即从“学习领域模型（LaDM）”范式转向“先学习预训练通用基础模型、再适配领域模型（LaGD）”的范式。在全新的LaGD范式下，过去十年间推动遥感图像智能理解发展的旧有数据集已不再适配全新的任务。我们认为，必须设计具备以下特征的新型数据集以支撑各类任务：1）泛化性：训练模型学习不同任务间的共享知识，以适配多样化任务；2）复杂场景理解：训练模型理解感兴趣目标的细粒度属性，并能够用自然语言描述场景；3）推理能力：训练模型具备实现高级视觉推理的能力。本文中，我们基于GPT-4V与现有数据集，构建了一个高质量、多样化的统一多模态指令跟随遥感图像理解数据集，命名为RS-GPT4V。为实现泛化性，我们采用由GPT-4V通过指令跟随推导生成的（问题、答案）格式，统一图像描述、定位等任务；为实现复杂场景理解，我们提出分层指令描述方法，其中局部策略用于描述目标的细粒度属性及其空间关系，全局策略则整合所有局部信息生成完整的指令描述；为实现推理能力，我们设计了多轮问答对以赋予模型推理能力。实验结果表明，经RS-GPT4V微调后的MLLM能够描述细粒度信息。该数据集可在以下地址获取：https://github.com/GeoX-Lab/RS-GPT4V。
英文摘要：
The remote sensing image intelligence understanding model is undergoing a new profound paradigm shift which has been promoted by multi-modal large language model (MLLM), i.e. from the paradigm learning a domain model (LaDM) shifts to paradigm learning a pre-trained general foundation model followed by an adaptive domain model (LaGD). Under the new LaGD paradigm, the old datasets, which have led to advances in RSI intelligence understanding in the last decade, are no longer suitable for fire-new tasks. We argued that a new dataset must be designed to lighten tasks with the following features: 1) Generalization: training model to learn shared knowledge among tasks and to adapt to different tasks; 2) Understanding complex scenes: training model to understand the fine-grained attribute of the objects of interest, and to be able to describe the scene with natural language; 3) Reasoning: training model to be able to realize high-level visual reasoning. In this paper, we designed a high-quality, diversified, and unified multimodal instruction-following dataset for RSI understanding produced by GPT-4V and existing datasets, which we called RS-GPT4V. To achieve generalization, we used a (Question, Answer) which was deduced from GPT-4V via instruction-following to unify the tasks such as captioning and localization; To achieve complex scene, we proposed a hierarchical instruction description with local strategy in which the fine-grained attributes of the objects and their spatial relationships are described and global strategy in which all the local information are integrated to yield detailed instruction descript; To achieve reasoning, we designed multiple-turn QA pair to provide the reasoning ability for a model. The empirical results show that the fine-tuned MLLMs by RS-GPT4V can describe fine-grained information. The dataset is available at: https://github.com/GeoX-Lab/RS-GPT4V.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
