arXiv 2401.097122024-01-18cs.CV

SkyEyeGPT：通过基于大型语言模型的指令微调统一遥感视觉-语言任务

SkyEyeGPT: Unifying Remote Sensing Vision-Language Tasks via Instruction Tuning with Large Language Model

Northwestern Polytechnical University（西北工业大学）；Technical University of Munich (TUM)（慕尼黑工业大学）

Yang Zhan, Zhitong Xiong, Yuan Yuan

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出遥感多模态大模型 SkyEyeGPT，利用 968k 高质量指令数据、对齐层与两阶段微调统一图像级和区域级视觉-语言任务，并在 8 个数据集上验证其优势。

摘要：

大型语言模型（LLM）最近已被扩展到视觉-语言领域，获得了令人瞩目的通用多模态能力。然而，面向遥感（RS）数据的多模态大型语言模型（MLLM）探索仍处于初期阶段，性能尚不能令人满意。在本工作中，我们提出 SkyEyeGPT，这是一个专为遥感视觉-语言理解设计的统一多模态大型语言模型。为此，我们精心构建了一个遥感多模态指令微调数据集，包含单任务和多任务对话指令。经过人工核验后，我们获得了一个包含 968k 个样本的高质量遥感指令遵循数据集。我们的研究表明，凭借简单而有效的设计，SkyEyeGPT 无需额外编码模块，即可在差异显著的任务上表现得出奇地好。具体而言，遥感视觉特征通过一个对齐层投影到语言域后，会与任务特定指令共同输入基于 LLM 的遥感解码器，以预测遥感开放式任务的答案。此外，我们设计了两阶段微调方法，以增强不同粒度下的指令遵循和多轮对话能力。在 8 个遥感视觉-语言任务数据集上的实验证明，SkyEyeGPT 在图像级和区域级任务（例如图像描述和视觉定位）上具有优越性。特别是在一些定性测试中，SkyEyeGPT 与 GPT-4V 相比展现出令人鼓舞的结果。在线演示、代码和数据集将在 https://github.com/ZhanYang-nwpu/SkyEyeGPT 发布。

英文摘要：

Large language models (LLMs) have recently been extended to the vision-language realm, obtaining impressive general multi-modal capabilities. However, the exploration of multi-modal large language models (MLLMs) for remote sensing (RS) data is still in its infancy, and the performance is not satisfactory. In this work, we introduce SkyEyeGPT, a unified multi-modal large language model specifically designed for RS vision-language understanding. To this end, we meticulously curate an RS multi-modal instruction tuning dataset, including single-task and multi-task conversation instructions. After manual verification, we obtain a high-quality RS instruction-following dataset with 968k samples. Our research demonstrates that with a simple yet effective design, SkyEyeGPT works surprisingly well on considerably different tasks without the need for extra encoding modules. Specifically, after projecting RS visual features to the language domain via an alignment layer, they are fed jointly with task-specific instructions into an LLM-based RS decoder to predict answers for RS open-ended tasks. In addition, we design a two-stage tuning method to enhance instruction-following and multi-turn dialogue ability at different granularities. Experiments on 8 datasets for RS vision-language tasks demonstrate SkyEyeGPT's superiority in image-level and region-level tasks, such as captioning and visual grounding. In particular, SkyEyeGPT exhibits encouraging results compared to GPT-4V in some qualitative tests. The online demo, code, and dataset will be released in https://github.com/ZhanYang-nwpu/SkyEyeGPT.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

AI总结：

摘要：

英文摘要：

相关论文