超越感知：通过多阶段任务评估抽象视觉推理
Beyond Perception: Evaluating Abstract Visual Reasoning through Multi-Stage Task
AI总结：
针对多模态大语言模型在抽象视觉推理中多阶段过程评估缺失的问题，提出基于RAVEN的多阶段基准MultiStAR及考虑中间步骤正确性的新指标MSEval，实验表明模型在复杂规则检测阶段仍面临挑战。
摘要：
当前的多模态大语言模型（MLLMs）在通用视觉推理方面表现出色，但在抽象视觉推理（AVR）方面仍缺乏深入探索，AVR需要高阶推理来识别超越简单感知的抽象规则。现有的AVR基准侧重于单步推理，强调最终结果，却忽略了推理过程的多阶段特性。以往研究发现MLLMs在这些基准上表现不佳，但未能解释其失败原因。为填补这一空白，我们引入了MultiStAR，一个基于RAVEN的多阶段AVR基准，旨在评估不同复杂度层级的推理能力。此外，现有的准确率等指标仅关注最终结果，未考虑中间步骤的正确性。因此，我们提出了一种新指标MSEval，在最终结果之外兼顾中间步骤的正确性。我们使用17个具有代表性的闭源和开源MLLMs在MultiStAR上进行了全面实验。结果表明，尽管现有MLLMs在基础感知任务上表现尚可，但在更复杂的规则检测阶段仍面临挑战。
英文摘要：
Current Multimodal Large Language Models (MLLMs) excel in general visual reasoning but remain underexplored in Abstract Visual Reasoning (AVR), which demands higher-order reasoning to identify abstract rules beyond simple perception. Existing AVR benchmarks focus on single-step reasoning, emphasizing the end result but neglecting the multi-stage nature of reasoning process. Past studies found MLLMs struggle with these benchmarks, but it doesn't explain how they fail. To address this gap, we introduce MultiStAR, a Multi-Stage AVR benchmark, based on RAVEN, designed to assess reasoning across varying levels of complexity. Additionally, existing metrics like accuracy only focus on the final outcomes while do not account for the correctness of intermediate steps. Therefore, we propose a novel metric, MSEval, which considers the correctness of intermediate steps in addition to the final outcomes. We conduct comprehensive experiments on MultiStAR using 17 representative close-source and open-source MLLMs. The results reveal that while existing MLLMs perform adequately on basic perception tasks, they continue to face challenges in more complex rule detection stages.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
