视觉推理基准：评估多模态大语言模型在小学课堂真实视觉问题上的能力
Visual Reasoning Benchmark: Evaluating Multimodal LLMs on Classroom-Authentic Visual Problems from Primary Education
AI总结：
本文提出视觉推理基准，用于评估多模态大语言模型在小学课堂真实视觉问题上的能力，揭示模型在静态与动态任务上的能力差异。
摘要：
人工智能模型在文本推理方面已取得最先进的成果；然而，它们在空间和关系结构上的推理能力仍然是一个关键瓶颈——特别是在早期年级数学中，这严重依赖于视觉。本文介绍了视觉推理基准（VRB），一个全新的数据集，旨在评估多模态大语言模型（MLLMs）在解决课堂真实视觉问题上的能力。该基准基于701个问题，这些问题是来自赞比亚和印度小学考试的，涵盖了类比推理、模式完成和空间匹配等任务。我们概述了该基准的方法和开发，其故意使用未经编辑的、文本最少的图像来测试模型是否能够满足小学教育的现实需求。我们的发现揭示了一个“崎岖前沿”的能力范围，其中模型在静态技能如计数和缩放方面表现出更好的熟练度，但面对动态操作如折叠、反射和旋转时却达到了一个明显的“空间上限”。这些弱点对课堂上视觉推理问题的使用构成了风险，可能导致错误评分、虚假支架和强化学生误解。因此，专注于教育的基准如VRB对于确定课堂中多模态工具的功能边界至关重要。
英文摘要：
AI models have achieved state-of-the-art results in textual reasoning; however, their ability to reason over spatial and relational structures remains a critical bottleneck -- particularly in early-grade maths, which relies heavily on visuals. This paper introduces the visual reasoning benchmark (VRB), a novel dataset designed to evaluate Multimodal Large Language Models (MLLMs) on their ability to solve authentic visual problems from classrooms. This benchmark is built on a set of 701 questions sourced from primary school examinations in Zambia and India, which cover a range of tasks such as reasoning by analogy, pattern completion, and spatial matching. We outline the methodology and development of the benchmark which intentionally uses unedited, minimal-text images to test if models can meet realistic needs of primary education. Our findings reveal a ``jagged frontier'' of capability where models demonstrate better proficiency in static skills such as counting and scaling, but reach a distinct ``spatial ceiling'' when faced with dynamic operations like folding, reflection, and rotation. These weaknesses pose a risk for classroom use on visual reasoning problems, with the potential for incorrect marking, false scaffolding, and reinforcing student misconceptions. Consequently, education-focused benchmarks like the VRB are essential for determining the functional boundaries of multimodal tools used in classrooms.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
