MagiC：评估面向基础视觉推理的多模态认知
MagiC: Evaluating Multimodal Cognition Toward Grounded Visual Reasoning
AI总结：
本文提出MagiC基准，通过评估答案正确性、推理有效性、基础保真度和自我纠正能力等维度，并引入新指标，全面评估大型视觉语言模型在基础视觉推理中的多模态认知能力，揭示现有方法的局限与机遇。
摘要：
大型视觉语言模型的最新进展在视觉问答和多模态推理中取得了令人瞩目的性能。然而，目前尚不清楚这些模型是真正执行了基础视觉推理，还是依赖于表面模式和数据集偏差。在本研究中，我们提出了 MagiC，这是一个旨在评估基础多模态认知的综合基准，它不仅评估答案准确性，还评估逐步推理的质量及其与相关视觉证据的对齐程度。该基准包含约 5,500 个由强模型输出生成的弱监督 QA 示例，以及 900 个带有细粒度标注的人工筛选示例，标注包括答案、理由和边界框基础。我们在四个维度上评估了参数规模从 7B 到 70B 的 15 个视觉语言模型：最终答案正确性、推理有效性、基础保真度和自我纠正能力。MagiC 还包含诊断设置，用于探测模型在对抗性视觉线索下的鲁棒性，并评估其内省式纠错能力。我们引入了 MagiScore 和 StepSense 等新指标，并提供了全面分析，揭示了当前基础视觉推理方法的关键局限性与机遇。
英文摘要：
Recent advances in large vision-language models have led to impressive performance in visual question answering and multimodal reasoning. However, it remains unclear whether these models genuinely perform grounded visual reasoning or rely on superficial patterns and dataset biases. In this work, we introduce MagiC, a comprehensive benchmark designed to evaluate grounded multimodal cognition, assessing not only answer accuracy but also the quality of step-by-step reasoning and its alignment with relevant visual evidence. Our benchmark includes approximately 5,500 weakly supervised QA examples generated from strong model outputs and 900 human-curated examples with fine-grained annotations, including answers, rationales, and bounding box groundings. We evaluate 15 vision-language models ranging from 7B to 70B parameters across four dimensions: final answer correctness, reasoning validity, grounding fidelity, and self-correction ability. MagiC further includes diagnostic settings to probe model robustness under adversarial visual cues and assess their capacity for introspective error correction. We introduce new metrics such as MagiScore and StepSense, and provide comprehensive analyses that reveal key limitations and opportunities in current approaches to grounded visual reasoning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
