STATUS Bench：用于评估视觉语言模型中物体状态理解的严谨基准
STATUS Bench: A Rigorous Benchmark for Evaluating Object State Understanding in Vision-Language Models
AI总结：
针对视觉语言模型难以精确识别物体状态的问题，提出首个严谨基准STATUS Bench，通过三项任务评估，并构建1300万描述的训练集，揭示现有模型零样本性能近乎随机，微调后Qwen2.5-VL可与Gemini 2.0 Flash媲美。
摘要：
物体状态识别旨在识别物体的具体状态，例如其位置状态（如打开或关闭）和功能状态（如开启或关闭）。尽管最近的视觉语言模型（VLMs）能够执行多种多模态任务，但它们究竟能在多大程度上精确识别物体状态仍不清楚。为缓解这一问题，我们引入了状态与转换理解基准（STATUS Bench），这是首个用于严格评估视觉语言模型在不同情境下理解物体状态细微变化能力的基准。具体而言，STATUS Bench引入了一种新颖的评估方案，要求视觉语言模型同时执行三项任务：物体状态识别（OSI）、图像检索（IR）和状态变化识别（SCI）。这些任务基于我们完全手工制作的数据集定义，该数据集包含图像对、对应的物体状态描述和状态变化描述。此外，我们引入了一个大规模训练数据集，即STATUS Train，该数据集包含1300万条半自动生成的描述，是促进该领域进一步研究的最大资源。在我们的实验中，我们证明了STATUS Bench能够实现严格的评估一致性，并揭示当前最先进的视觉语言模型在捕捉细微物体状态差异方面仍存在显著困难。令人惊讶的是，在提出的严格评估方案下，大多数开放权重视觉语言模型的零样本性能仅处于随机水平。在STATUS Train上进行微调后，Qwen2.5-VL达到了与Gemini 2.0 Flash相当的性能。这些发现强调了STATUS Bench和STATUS Train在推进视觉语言模型研究中物体状态识别领域的必要性。
英文摘要：
Object state recognition aims to identify the specific condition of objects, such as their positional states (e.g., open or closed) and functional states (e.g., on or off). While recent Vision-Language Models (VLMs) are capable of performing a variety of multimodal tasks, it remains unclear how precisely they can identify object states. To alleviate this issue, we introduce the STAte and Transition UnderStanding Benchmark (STATUS Bench), the first benchmark for rigorously evaluating the ability of VLMs to understand subtle variations in object states in diverse situations. Specifically, STATUS Bench introduces a novel evaluation scheme that requires VLMs to perform three tasks simultaneously: object state identification (OSI), image retrieval (IR), and state change identification (SCI). These tasks are defined over our fully hand-crafted dataset involving image pairs, their corresponding object state descriptions and state change descriptions. Furthermore, we introduce a large-scale training dataset, namely STATUS Train, which consists of 13 million semi-automatically created descriptions. This dataset serves as the largest resource to facilitate further research in this area. In our experiments, we demonstrate that STATUS Bench enables rigorous consistency evaluation and reveal that current state-of-the-art VLMs still significantly struggle to capture subtle object state distinctions. Surprisingly, under the proposed rigorous evaluation scheme, most open-weight VLMs exhibited chance-level zero-shot performance. After fine-tuning on STATUS Train, Qwen2.5-VL achieved performance comparable to Gemini 2.0 Flash. These findings underscore the necessity of STATUS Bench and Train for advancing object state recognition in VLM research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
