arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.05091cs.CL
收藏

MM-MATH:通过过程评估与细粒度分类推进多模态数学评估

MM-MATH: Advancing Multimodal Math Evaluation with Process Evaluation and Fine-grained Classification

Tsinghua University(清华大学)

Kai Sun, Yushi Bai, Ji Qi, Lei Hou, Juanzi Li

AI总结:

本文提出含5,929道视觉中学数学题的 MM-MATH 基准,结合结果与过程评估及细粒度分类,揭示十个 LMM 与人类在多模态数学推理上的显著差距。

摘要:

为推进大型多模态模型(LMM)多模态数学推理能力的评估,本文提出一个新型基准 MM-MATH。该基准包含5,929道带有视觉情境的开放式中学数学题,并按难度、年级和知识点进行细粒度分类。与依赖二元答案比较的现有基准不同,MM-MATH同时纳入结果评估和过程评估。过程评估采用 LMM-as-a-judge(LMM担任评判者)自动分析解题步骤,识别错误并将其归入具体错误类型。在 MM-MATH 上对十个模型进行的广泛评估揭示了现有 LMM 面临的重大挑战,突出表现为它们对视觉信息的利用有限,且在更高难度题目上存在困难。表现最好的模型在 MM-MATH 上仅达到31%的准确率,而人类为82%。这表明该基准对现有模型具有挑战性,也说明当前模型与人类在多模态推理能力上存在显著差距。过程评估还显示,图表误解是最常见错误,占全部错误案例的一半以上,凸显多模态推理中提升图像理解能力的必要性。

英文摘要:

To advance the evaluation of multimodal math reasoning in large multimodal models (LMMs), this paper introduces a novel benchmark, MM-MATH. MM-MATH consists of 5,929 open-ended middle school math problems with visual contexts, with fine-grained classification across difficulty, grade level, and knowledge points. Unlike existing benchmarks relying on binary answer comparison, MM-MATH incorporates both outcome and process evaluations. Process evaluation employs LMM-as-a-judge to automatically analyze solution steps, identifying and categorizing errors into specific error types. Extensive evaluation of ten models on MM-MATH reveals significant challenges for existing LMMs, highlighting their limited utilization of visual information and struggles with higher-difficulty problems. The best-performing model achieves only 31% accuracy on MM-MATH, compared to 82% for humans. This highlights the challenging nature of our benchmark for existing models and the significant gap between the multimodal reasoning capabilities of current models and humans. Our process evaluation reveals that diagram misinterpretation is the most common error, accounting for more than half of the total error cases, underscoring the need for improved image comprehension in multimodal reasoning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑