VISTAQA: 评估联合视觉问答与像素级证据
VISTAQA: Benchmarking Joint Visual Question Answering and Pixel-Level Evidence
AI总结：
本文提出VISTAQA基准，用于评估视觉问答中自由回答的正确性和像素级证据的定位，通过引入GROVE指标，强调回答正确性与视觉证据对齐的重要性，实验显示现有系统在该指标下表现有限，揭示了回答准确性和视觉证据对齐之间的显著差距。
摘要：
建立模型预测与支持它们的视觉证据之间的清晰联系对于多模态推理的透明性和可靠性至关重要，但当前的多模态大语言模型（MLLM）评估并未明确强制这种对齐。现有的基准评估要么单独评估文本答案的正确性，要么单独评估像素级定位，使推理与定位的耦合成为一个开放性挑战。我们介绍了VISTAQA，一个用于联合评估自由回答正确性和像素级证据定位的全面基准。VISTAQA包含1,157个专家整理的样本，涵盖六种任务类型和六个视觉领域，从直接感知到组合和关系推理。VISTAQA要求模型不仅要正确回答，还要提供精确的分割掩码以支持其答案。它还包含有幻觉意识的例子，其中不存在有效的视觉证据。为了支持这种增强的评估，我们引入了GROVE，一个统一的评估指标，通过每样本几何均值结合文本准确性与定位质量，确保两者都不能补偿对方的不足。在接地意识模型和混合管道与通用MLLM的全面实验中，即使最强的系统在GROVE下也表现有限，突显了回答准确性和视觉证据对齐之间的显著差距。
英文摘要：
Establishing a clear link between model predictions and the visual evidence that supports them is critical for transparency and reliability in multimodal reasoning, yet current multimodal large language model (MLLM) evaluations do not explicitly enforce this alignment. Existing benchmarks assess either textual answer correctness or pixel-level localization in isolation, leaving the coupling of reasoning and grounding an open challenge. We introduce VISTAQA, a comprehensive benchmark for joint evaluation of free-form answer correctness and pixel-level evidence grounding in visual question answering. VISTAQA comprises 1,157 expert-curated samples spanning six task types and six visual domains, ranging from direct perception to compositional and relational reasoning. VISTAQA requires models to not only answer correctly, but to also provide precise segmentation masks that support their answers. It also includes hallucination-aware examples where no valid visual evidence exists. To support this enhanced evaluation, we introduce GROVE, a unified evaluation metric that enforces joint correctness by combining textual accuracy and grounding quality via a per-sample geometric mean, ensuring neither dimension can compensate for deficiencies in the other. Comprehensive experiments across grounding-aware models and hybrid pipelines with general-purpose MLLMs reveal that even the strongest systems achieve limited performance under GROVE, highlighting a substantial gap between answer accuracy and visual evidence alignment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
