arXiv 2504.127952025-11-06cs.CV

EarthGPT-X：一种基于视觉提示的多层级多源遥感影像理解空间多模态大语言模型

EarthGPT-X: A Spatial MLLM for Multi-level Multi-Source Remote Sensing Imagery Understanding with Visual Prompting

School of Interdisciplinary Science, Beijing Institute of Technology（交叉科学学院，北京理工大学）；College of Computing and Data Science, Nanyang Technological University（计算与数据科学学院，南洋理工大学）；National Key Laboratory of Science and Technology on Space-Born Intelligent Information Processing, Beijing Institute of Technology（空间智能信息处理国家重点实验室，北京理工大学）；School of Optics and Photonics, Beijing Institute of Technology（光学与 photonics 学院，北京理工大学）；State Key Laboratory of Explosion Science and Safety Protection, Beijing（爆炸科学与安全防护国家重点实验室，北京）

Wei Zhang, Miaoxin Cai, Yaqian Ning, Tong Zhang, Yin Zhuang, Shijian Lu, He Chen, Jun Li, Xuerui Mao

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

EarthGPT-X 提出首个统一多源遥感影像理解的空间多模态大语言模型，通过双提示机制、多源多层级提示数据集和跨域单阶段融合训练，在单一框架中实现多源、多任务、多层级的视觉提示解译。

摘要：

自然域多模态大语言模型（MLLMs）的最新进展已经证明了通过视觉和文本提示进行有效空间推理的能力。然而，将其直接迁移到遥感（RS）领域受到异构传感物理特性、多样模态和独特空间尺度的阻碍。现有的遥感多模态大语言模型主要局限于光学影像和纯语言交互，难以实现灵活且可扩展的实际应用。本文提出了 EarthGPT-X，这是首个灵活的空间多模态大语言模型，在单一框架中统一多源遥感影像理解，并在多样视觉提示下完成粗粒度和细粒度视觉任务。与先前模型不同，EarthGPT-X 引入了：1）一种双提示机制，将文本指令与各种视觉提示（即点、框和自由形式）相结合，以模拟人类生活中指称的多样性；2）一个全面的多源多层级提示数据集，使模型超越整体影像理解，支持层级化空间推理，包括场景级理解以及细粒度目标属性和关系分析；3）一种跨域单阶段融合训练策略，实现跨模态和跨任务的高效且一致的对齐。大量实验表明，EarthGPT-X 显著优于先前的自然域和遥感多模态大语言模型，建立了首个能够在遥感场景中使用视觉提示进行多源、多任务和多层级解译的框架。

英文摘要：

Recent advances in natural-domain multi-modal large language models (MLLMs) have demonstrated effective spatial reasoning through visual and textual prompting. However, their direct transfer to remote sensing (RS) is hindered by heterogeneous sensing physics, diverse modalities, and unique spatial scales. Existing RS MLLMs are mainly limited to optical imagery and plain language interaction, preventing flexible and scalable real-world applications. In this article, EarthGPT-X is proposed, the first flexible spatial MLLM that unifies multi-source RS imagery comprehension and accomplishes both coarse-grained and fine-grained visual tasks under diverse visual prompts in a single framework. Distinct from prior models, EarthGPT-X introduces: 1) a dual-prompt mechanism combining text instructions with various visual prompts (i.e., point, box, and free-form) to mimic the versatility of referring in human life; 2) a comprehensive multi-source multi-level prompting dataset, the model advances beyond holistic image understanding to support hierarchical spatial reasoning, including scene-level understanding and fine-grained object attributes and relational analysis; 3) a cross-domain one-stage fusion training strategy, enabling efficient and consistent alignment across modalities and tasks. Extensive experiments demonstrate that EarthGPT-X substantially outperforms prior nature and RS MLLMs, establishing the first framework capable of multi-source, multi-task, and multi-level interpretation using visual prompting in RS scenarios.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。