AI 中文总结
本文提出OVEarth-Bench基准，从类别广度与查询多样性两方面扩展开放词汇地球观测评估，经实验发现MLLM类方法性能最优，为该领域未来研究提供指导。
AI 中文摘要
开放词汇地球观测（EO）旨在定位自然语言指定的地理空间概念，而非固定标签集。现有基准通常类别词汇较窄或查询形式有限，为填补这一空白，本文提出OVEarth-Bench，从两方面扩展现有评估：一是类别广度，通过包含正负表述的广泛分层类别覆盖实现；二是查询多样性，涵盖词汇、指称及推理查询。该基准支持统一零样本协议下的掩码与框定位，本文评估了一系列通用方法及EO专用方法。评估揭示：（1）当前方法性能仍有限，更广泛的类别覆盖可使模型排名更稳定；（2）基于多模态大语言模型（MLLM）的方法整体性能最强；（3）EO专用方法通常性能弱于通用模型，极少达到最优方法水平。这些发现为未来开放词汇EO方法的设计提供指导，并强调开发更真实、多样、高质量及大规模基准以实现可靠评估的重要性。本文的数据集与评估包已发布在该httpsURL。
英文摘要
Open-vocabulary Earth observation (EO) aims to localize geospatial concepts specified in natural language rather than a fixed label set. Existing benchmarks, however, usually cover narrow category vocabularies or limited query forms. To fill this gap, we introduce OVEarth-Bench, which extends existing evaluation in two directions: category breadth, through broad hierarchical category coverage with positive and negative expressions, and query diversity, through vocabulary, referring, and reasoning queries. The benchmark supports mask and box localization under a unified zero-shot protocol. We evaluate a broad set of general and EO-specific methods. The evaluation reveals that: (1) the performance of current methods remains limited, while broader category coverage yields more stable model rankings; (2) MLLM-based methods achieve the strongest overall performance; and (3) EO-specific methods generally underperform general models and rarely match the strongest methods. These findings provide guidance for future open-vocabulary EO method design and highlight the importance of developing more realistic, diverse, high-quality, and large-scale benchmarks for reliable evaluation. Our data and evaluation package are released at https://earth-insights.github.io/OVEarth-bench.
