Box-QAymo：面向自动驾驶的框指涉视觉问答数据集
Box-QAymo: Box-Referring VQA Dataset for Autonomous Driving
AI总结：
本文提出Box-QAymo框指涉VQA数据集与分层评估协议，通过用户绘制边界框的交互方式，系统评估并微调视觉语言模型在自动驾驶场景中对指定对象的属性、运动及跨帧时空推理能力，揭示了现有模型在真实场景感知问答中的显著不足。
摘要：
可解释的通信对于安全可信的自动驾驶至关重要，然而当前的视觉语言模型（VLMs）通常在理想化假设下运行，难以在真实场景中捕捉用户意图。现有的面向驾驶的VQA数据集仅限于全场景描述或航点预测，无法评估VLMs能否响应用户指定的局部查询。我们引入了Box-QAymo，一个框指涉数据集和基准，旨在评估和微调VLMs对用户指定对象的时空推理能力。用户通过绘制边界框来表达意图，为复杂场景中的聚焦查询提供了快速直观的交互方式。具体而言，我们提出了一种分层评估协议，从二元的健全性检查问题开始以评估模型的基本能力，并逐步推进到（1）框指涉对象的属性预测，（2）目标实例的运动理解，以及（3）跨帧对象间动态的时空运动推理。为此，我们众包了细粒度对象类别和视觉属性，以反映驾驶员所面临的复杂性，并提取对象轨迹来构建具有时间依据的问答对。通过负采样、时间一致性检查和难度感知平衡进行的严格质量控制，保证了数据集的鲁棒性和多样性。我们的综合评估揭示了当前VLMs在被问及感知问题时的显著局限性，凸显了实现真实世界性能方面的差距。这项工作为开发更鲁棒、可解释的自动驾驶系统奠定了基础，使其能够在真实世界条件下与用户有效通信。项目页面和数据集可在 https://djamahl99.github.io/qaymo-pages/ 获取。
英文摘要：
Interpretable communication is essential for safe and trustworthy autonomous driving, yet current vision-language models (VLMs) often operate under idealized assumptions and struggle to capture user intent in real-world scenarios. Existing driving-oriented VQA datasets are limited to full-scene descriptions or waypoint prediction, preventing the assessment of whether VLMs can respond to localized user-driven queries. We introduce Box-QAymo, a box-referring dataset and benchmark designed to both evaluate and finetune VLMs on spatial and temporal reasoning over user-specified objects. Users express intent by drawing bounding boxes, offering a fast and intuitive interface for focused queries in complex scenes. Specifically, we propose a hierarchical evaluation protocol that begins with binary sanity-check questions to assess basic model capacities, and progresses to (1) attribute prediction for box-referred objects, (2) motion understanding of target instances, and (3) spatiotemporal motion reasoning over inter-object dynamics across frames. To support this, we crowd-sourced fine-grained object classes and visual attributes that reflect the complexity drivers encounter, and extract object trajectories to construct temporally grounded QA pairs. Rigorous quality control through negative sampling, temporal consistency checks, and difficulty-aware balancing guarantee dataset robustness and diversity. Our comprehensive evaluation reveals significant limitations in current VLMs when queried about perception questions, highlighting the gap in achieving real-world performance. This work provides a foundation for developing more robust and interpretable autonomous driving systems that can communicate effectively with users under real-world conditions. Project page and dataset are available at https://djamahl99.github.io/qaymo-pages/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
