Point-It-Out：在多阶段视觉定位中对视觉语言模型的具身推理进行基准测试
Point-It-Out: Benchmarking Embodied Reasoning for Vision Language Models in Multi-Stage Visual Grounding
AI总结：
提出Point-It-Out三阶段视觉定位基准，系统评估十余个VLM的具身推理能力，发现强通用模型在精确定位和轨迹规划上存在短板。
摘要：
视觉语言模型（VLM）已在广泛任务中展现出令人瞩目的世界知识，因此成为具身推理应用的有力候选。然而，现有基准主要通过基于图像标注的多项选择题来评估VLM的具身推理能力，例如选择哪条轨迹更能描述图像中的事件。在本工作中，我们提出Point-It-Out（PIO）基准，这是一个旨在通过精确视觉定位系统评估VLM具身推理能力的新型基准。我们提出了跨越三个阶段的分层评估协议（S1：指称对象定位，S2：任务驱动指向，S3：视觉轨迹预测），数据采集自具身智能的关键领域，包括室内、厨房、驾驶和机器人操作场景。对十余个最先进VLM进行的大量实验揭示了若干有趣发现。例如，尽管GPT-4o等强大的通用模型在许多基准（如语言、感知和推理）上表现优异，但在精确视觉定位方面却不如部分开源模型；MoLMO等模型在S1和S2中表现良好，但在需要将定位与视觉轨迹规划相结合的S3中遇到困难。
英文摘要：
Vision-Language Models (VLMs) have demonstrated impressive world knowledge across a wide range of tasks, making them promising candidates for embodied reasoning applications. However, existing benchmarks primarily evaluate the embodied reasoning ability of VLMs through multiple-choice questions based on image annotations -- for example, selecting which trajectory better describes an event in the image. In this work, we introduce the Point-It-Out (PIO) benchmark, a novel benchmark designed to systematically assess the embodied reasoning abilities of VLMs through precise visual grounding. We propose a hierarchical evaluation protocol spanning three stages (S1: referred-object localization, S2: task-driven pointing, and S3: visual trace prediction), with data collected from critical domains for embodied intelligence, including indoor, kitchen, driving, and robotic manipulation scenarios. Extensive experiments with over ten state-of-the-art VLMs reveal several interesting findings. For example, strong general-purpose models such as GPT-4o, while excelling on many benchmarks (e.g., language, perception, and reasoning), underperform compared to some open-source models in precise visual grounding; models such as MoLMO perform well in S1 and S2 but struggle in S3, where requires grounding combined with visual trace planning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
