GeoChat：用于遥感的基础大型视觉语言模型
GeoChat: Grounded Large Vision-Language Model for Remote Sensing
AI总结：
针对通用视觉语言模型在遥感场景表现不佳的问题，提出首个多功能遥感VLM模型GeoChat，通过构建专属多模态指令数据集，实现高分辨率图像的区域级推理、视觉定位对话等多任务零样本能力。
摘要：
大型视觉语言模型（VLMs）的最新进展在自然图像领域展现出巨大前景，允许用户就给定的视觉内容进行对话。然而，此类通用领域的VLMs在遥感（RS）场景下表现不佳，在面临遥感领域特定查询时会导致不准确或捏造的信息。这种行为的出现是由于遥感影像引入了独特的挑战。例如，为了处理具有跨类别多样尺度变化和许多小对象的高分辨率遥感影像，除了整体场景解读外，还需要进行区域级推理。此外，缺乏特定领域的多模态指令跟随数据以及强大的遥感骨干模型，使得模型难以将其行为与用户查询对齐。为了解决这些局限性，我们提出了GeoChat——第一个多功能遥感VLM，它提供了高分辨率遥感图像的多任务对话能力。具体而言，GeoChat不仅能回答图像级查询，还接受区域输入以进行特定区域的对话。此外，它可以通过引用对象的空间坐标，在其响应中视觉地定位对象。为了解决缺乏特定领域数据集的问题，我们通过扩展现有各种遥感数据集中的图像-文本对，生成了一个新颖的遥感多模态指令跟随数据集。我们建立了一个全面的遥感多任务对话基准，并与多种基线方法进行了比较。GeoChat在各种遥感任务上展示了稳健的零样本性能，例如图像和区域描述、视觉问答、场景分类、视觉定位对话和指代检测。我们的代码可在https://github.com/mbzuai-oryx/geochat获取。
英文摘要：
Recent advancements in Large Vision-Language Models (VLMs) have shown great promise in natural image domains, allowing users to hold a dialogue about given visual content. However, such general-domain VLMs perform poorly for Remote Sensing (RS) scenarios, leading to inaccurate or fabricated information when presented with RS domain-specific queries. Such a behavior emerges due to the unique challenges introduced by RS imagery. For example, to handle high-resolution RS imagery with diverse scale changes across categories and many small objects, region-level reasoning is necessary alongside holistic scene interpretation. Furthermore, the lack of domain-specific multimodal instruction following data as well as strong backbone models for RS make it hard for the models to align their behavior with user queries. To address these limitations, we propose GeoChat - the first versatile remote sensing VLM that offers multitask conversational capabilities with high-resolution RS images. Specifically, GeoChat can not only answer image-level queries but also accepts region inputs to hold region-specific dialogue. Furthermore, it can visually ground objects in its responses by referring to their spatial coordinates. To address the lack of domain-specific datasets, we generate a novel RS multimodal instruction-following dataset by extending image-text pairs from existing diverse RS datasets. We establish a comprehensive benchmark for RS multitask conversations and compare with a number of baseline methods. GeoChat demonstrates robust zero-shot performance on various RS tasks, e.g., image and region captioning, visual question answering, scene classification, visually grounded conversations and referring detection. Our code is available at https://github.com/mbzuai-oryx/geochat.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
