擅长字幕生成，不擅计数：在地球观测数据上基准测试GPT-4V
Good at captioning, bad at counting: Benchmarking GPT-4V on Earth observation data
AI总结：
本研究针对大视觉语言模型在地球观测数据上的能力迁移问题，构建涵盖多应用场景的综合基准测试，发现GPT-4V等模型开放式任务表现好但空间推理弱，且将公开基准助力相关评估。
摘要：
大视觉语言模型（VLM）在涉及视觉输入与自然语言指令的复杂任务上展现出了出色性能。然而，目前尚不清楚其在自然图像上的能力能在多大程度上迁移至地球观测（EO）数据——这类数据以卫星和航空图像为主，在VLM训练数据中较为少见。本研究中，我们提出了一个综合基准，通过评估VLM在场景理解、定位与计数以及变化检测任务上的能力，衡量其成为EO数据实用工具的进展。受实际应用启发，我们的基准涵盖城市监测、救灾、土地利用和生态保护等场景。我们发现，尽管GPT-4V等最先进的VLM拥有丰富的世界知识，在位置理解、图像字幕生成等开放式任务上表现强劲，但其空间推理能力薄弱，限制了其在目标定位与计数任务上的实用性。我们的基准将公开发布于https://vleo.danielz.ch/ 以及Hugging Face平台的https://huggingface.co/collections/mit-ei/vleo-benchmark-datasets-65b789b0466555489cce0d70，以便开展模型评估。
英文摘要：
Large Vision-Language Models (VLMs) have demonstrated impressive performance on complex tasks involving visual input with natural language instructions. However, it remains unclear to what extent capabilities on natural images transfer to Earth observation (EO) data, which are predominantly satellite and aerial images less common in VLM training data. In this work, we propose a comprehensive benchmark to gauge the progress of VLMs toward being useful tools for EO data by assessing their abilities on scene understanding, localization and counting, and change detection tasks. Motivated by real-world applications, our benchmark includes scenarios like urban monitoring, disaster relief, land use, and conservation. We discover that, although state-of-the-art VLMs like GPT-4V possess extensive world knowledge that leads to strong performance on open-ended tasks like location understanding and image captioning, their poor spatial reasoning limits usefulness on object localization and counting tasks. Our benchmark will be made publicly available at https://vleo.danielz.ch/ and on Hugging Face at https://huggingface.co/collections/mit-ei/vleo-benchmark-datasets-65b789b0466555489cce0d70 for easy model evaluation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
