MV-MATH：在多视觉语境下评估多模态数学推理能力
MV-MATH: Evaluating Multimodal Math Reasoning in Multi-Visual Contexts
AI总结：
针对现有多模态数学基准多为单视觉语境、与真实多视觉应用场景脱节的问题，研究者推出包含2009道K-12阶段多图穿插文本数学题的MV-MATH基准，实验发现MLLMs在该基准上表现远逊于人类，并分析了相关性能与错误模式。
摘要：
多模态大语言模型（MLLMs）已在各类数据集的视觉语境数学推理任务中展现出可观能力。但现有多数多模态数学基准仅局限于单视觉语境，与现实世界数学应用中常见的多视觉场景存在偏差。为填补这一空白，我们推出MV-MATH：一个经精心整理的包含2009道高质量数学题的数据集。每道题均融合多张与文本穿插排布的图像，源自真实的K-12教学场景，并配有详细标注。MV-MATH涵盖选择题、自由作答题和多步骤问题，覆盖11个学科领域、3个难度等级，可作为评估MLLMs在多视觉语境下数学推理能力的全面且严谨的基准。通过大量实验，我们发现MLLMs在多视觉数学任务中面临显著挑战，在MV-MATH上的表现与人类能力存在较大差距。此外，我们还分析了多种模型的表现与错误模式，为理解多视觉场景下MLLMs的数学推理能力提供了洞见。
英文摘要：
Multimodal Large Language Models (MLLMs) have shown promising capabilities in mathematical reasoning within visual contexts across various datasets. However, most existing multimodal math benchmarks are limited to single-visual contexts, which diverges from the multi-visual scenarios commonly encountered in real-world mathematical applications. To address this gap, we introduce MV-MATH: a meticulously curated dataset of 2,009 high-quality mathematical problems. Each problem integrates multiple images interleaved with text, derived from authentic K-12 scenarios, and enriched with detailed annotations. MV-MATH includes multiple-choice, free-form, and multi-step questions, covering 11 subject areas across 3 difficulty levels, and serves as a comprehensive and rigorous benchmark for assessing MLLMs' mathematical reasoning in multi-visual contexts. Through extensive experimentation, we observe that MLLMs encounter substantial challenges in multi-visual math tasks, with a considerable performance gap relative to human capabilities on MV-MATH. Furthermore, we analyze the performance and error patterns of various models, providing insights into MLLMs' mathematical reasoning capabilities within multi-visual settings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
