OmniEarth：一个评估遥感任务中视觉-语言模型的基准
OmniEarth: A Benchmark for Evaluating Vision-Language Models in Geospatial Tasks
AI总结：
OmniEarth是一个评估遥感任务中视觉-语言模型性能的基准，通过多维任务和严格测试协议，揭示现有模型在复杂地理任务中的不足。
摘要：
视觉-语言模型（VLMs）在通用领域任务中展示了有效的感知和推理能力，推动了其在地球观测中的应用。然而，系统性评估遥感视觉-语言模型（RSVLMs）的基准仍然缺乏。为解决这一差距，我们引入了OmniEarth，一个在现实地球观测场景中评估RSVLMs的基准。OmniEarth沿三个能力维度组织任务：感知、推理和鲁棒性。它定义了28个细粒度任务，涵盖多源传感数据和多样的地理空间上下文。该基准支持两种任务形式：多选VQA和开放性VQA。后者包括纯文本输出用于描述任务、边界框输出用于视觉定位任务以及掩码输出用于分割任务。为了减少语言偏差并检验模型预测是否依赖于视觉证据，OmniEarth采用盲测协议和五重语义一致性要求。OmniEarth包括9275张经过精心质量控制的图像，包括吉林一号（JL-1）的专有卫星图像，以及44210条人工验证的指令。我们对基于对比学习的模型、通用闭源和开源VLMs以及RSVLMs进行了系统评估。结果显示，现有VLMs在地理空间复杂任务中仍存在困难，揭示了需要解决的明显差距，以适用于遥感应用。OmniEarth可在https://huggingface.co/datasets/sjeeudd/OmniEarth上公开获取。
英文摘要：
Vision-Language Models (VLMs) have demonstrated effective perception and reasoning capabilities on general-domain tasks, leading to growing interest in their application to Earth observation. However, a systematic benchmark for comprehensively evaluating remote sensing vision-language models (RSVLMs) remains lacking. To address this gap, we introduce OmniEarth, a benchmark for evaluating RSVLMs under realistic Earth observation scenarios. OmniEarth organizes tasks along three capability dimensions: perception, reasoning, and robustness. It defines 28 fine-grained tasks covering multi-source sensing data and diverse geospatial contexts. The benchmark supports two task formulations: multiple-choice VQA and open-ended VQA. The latter includes pure text outputs for captioning tasks, bounding box outputs for visual grounding tasks, and mask outputs for segmentation tasks. To reduce linguistic bias and examine whether model predictions rely on visual evidence, OmniEarth adopts a blind test protocol and a quintuple semantic consistency requirement. OmniEarth includes 9,275 carefully quality-controlled images, including proprietary satellite imagery from Jilin-1 (JL-1), along with 44,210 manually verified instructions. We conduct a systematic evaluation of contrastive learning-based models, general closed-source and open-source VLMs, as well as RSVLMs. Results show that existing VLMs still struggle with geospatially complex tasks, revealing clear gaps that need to be addressed for remote sensing applications. OmniEarth is publicly available at https://huggingface.co/datasets/sjeeudd/OmniEarth.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
