使用 MATH-Vision 数据集衡量多模态数学推理能力
Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset
AI总结：
针对现有视觉数学推理基准多样性和学科覆盖不足的问题，本文提出包含 3,040 道真实竞赛题、覆盖 16 个学科和 5 个难度等级的 MATH-Vision 数据集，并通过实验揭示了当前多模态模型与人类水平之间的显著差距。
摘要：
大型多模态模型（LMMs）的最新进展在视觉情境下的数学推理中展现出令人鼓舞的成果，部分模型在 MathVista 等现有基准上已接近人类水平。然而，我们观察到这些基准在问题多样性和学科覆盖广度上存在显著局限。为解决这一问题，我们提出了 MATH-Vision（MATH-V）数据集，这是一个精心整理的包含 3,040 道高质量视觉情境数学题目的集合，题目来源于真实数学竞赛。该数据集涵盖 16 个不同的数学学科，并按 5 个难度等级进行划分，为评估 LMMs 的数学推理能力提供了全面且多样化的挑战集。通过大量实验，我们揭示了当前 LMMs 与人类在 MATH-V 上的表现之间存在显著差距，凸显了进一步推进 LMMs 发展的迫切性。此外，我们细致的分类体系支持对 LMMs 进行深入的错误分析，为未来研究与开发提供了有价值的指导。项目地址为 https://mathvision-cuhk.github.io。
英文摘要：
Recent advancements in Large Multimodal Models (LMMs) have shown promising results in mathematical reasoning within visual contexts, with models approaching human-level performance on existing benchmarks such as MathVista. However, we observe significant limitations in the diversity of questions and breadth of subjects covered by these benchmarks. To address this issue, we present the MATH-Vision (MATH-V) dataset, a meticulously curated collection of 3,040 high-quality mathematical problems with visual contexts sourced from real math competitions. Spanning 16 distinct mathematical disciplines and graded across 5 levels of difficulty, our dataset provides a comprehensive and diverse set of challenges for evaluating the mathematical reasoning abilities of LMMs. Through extensive experimentation, we unveil a notable performance gap between current LMMs and human performance on MATH-V, underscoring the imperative for further advancements in LMMs. Moreover, our detailed categorization allows for a thorough error analysis of LMMs, offering valuable insights to guide future research and development. The project is available at https://mathvision-cuhk.github.io
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
