你的模型真的是优秀的数学推理器吗？用清单法评估数学推理能力
Is Your Model Really A Good Math Reasoner? Evaluating Mathematical Reasoning with Checklist
AI总结：
针对现有数学推理基准易过拟合、无法真实反映模型能力的问题，本文提出评估清单MathCheck及自动生成工具，构建文本和多模态升级基准，测试26个LLM和17个MLLM，证明其能更准确衡量数学推理能力且可扩展到其他推理任务。
摘要：
卓越的数学推理能力是证明大语言模型（LLM）强大性能的核心特征之一。如何全面定义并评估大语言模型的数学能力，甚至反映真实场景下的用户体验，已成为一个关键问题。当前的基准测试主要聚焦于解题能力，存在严重的模型过拟合风险，无法准确衡量模型真正的数学推理能力。本文提出，如果模型真正理解了问题，它应当能在多种不同任务中稳定发挥作用。为此，我们推出了MathCheck——一个精心设计的、用于测试任务泛化能力和推理鲁棒性的清单，以及一套可高效生成清单的自动化工具。MathCheck包含多项数学推理任务和鲁棒性测试，可同时对数学推理能力和行为测试进行综合评估。借助MathCheck，我们构建了MathCheck-GSM和MathCheck-GEO，分别用于评估数学文本推理能力和多模态推理能力，它们是GSM8k、GeoQA、UniGeo和Geometry3K等基准的升级版本。我们使用MathCheck-GSM和MathCheck-GEO评估了26个大语言模型和17个多模态大语言模型（MLLM）。结果显示，尽管GPT-4o等前沿大语言模型在清单的各项能力测试中依然表现优异，但许多其他模型家族的性能出现了显著下降。进一步实验表明，与传统数学基准相比，MathCheck能更准确地反映真实数学能力，且与数学智能的对应关系更具线性特征，从而验证了我们的设计思路。利用MathCheck，我们可以高效开展有价值的行为分析，对模型进行深度研究。最后，我们证明这种清单范式可以轻松扩展到其他推理任务。
英文摘要：
Exceptional mathematical reasoning ability is one of the key features that demonstrate the power of large language models (LLMs). How to comprehensively define and evaluate the mathematical abilities of LLMs, and even reflect the user experience in real-world scenarios, has emerged as a critical issue. Current benchmarks predominantly concentrate on problem-solving capabilities, presenting a substantial risk of model overfitting and fails to accurately measure the genuine mathematical reasoning abilities. In this paper, we argue that if a model really understands a problem, it should be robustly applied across a diverse array of tasks. To this end, we introduce MathCheck, a well-designed checklist for testing task generalization and reasoning robustness, as well as an automatic tool to generate checklists efficiently. MathCheck includes multiple mathematical reasoning tasks and robustness tests to facilitate a comprehensive evaluation of both mathematical reasoning ability and behavior testing. Utilizing MathCheck, we develop MathCheck-GSM and MathCheck-GEO to assess math textual reasoning and multi-modal reasoning abilities, respectively, serving as upgraded versions of benchmarks including GSM8k, GeoQA, UniGeo, and Geometry3K. We adopt MathCheck-GSM and MathCheck-GEO to evaluate 26 LLMs and 17 MLLMs. Our results demonstrate that while frontier LLMs like GPT-4o continue to excel in various abilities on the checklist, many other model families exhibit a significant decline. Further experiments indicate that, compared to traditional math benchmarks, MathCheck better reflects true mathematical abilities and represents mathematical intelligence more linearly, thereby supporting our design. Using MathCheck, we can efficiently conduct informative behavior analysis to deeply investigate models. Finally, we show that our checklist paradigm can easily extend to other reasoning tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
