UTMath：通过“推理到编码”思维实现单元测试式数学评估
UTMath: Math Evaluation with Unit Test via Reasoning-to-Coding Thoughts
AI总结：
针对现有数学推理基准评估局限，本文提出含1053道多领域题的UTMath基准与RCoT推理方法，同时发布7万+样本训练集，为大语言模型数学能力评估与提升提供支撑。
摘要：
数学推理能力的评估对于推动通用人工智能（AGI）发展至关重要。尽管大语言模型（LLM）在求解数学问题方面已展现出令人瞩目的性能，但GSM8K、MATH等现有基准存在局限性：问题定义狭窄，仅包含特定数字，且依赖预设规则，这阻碍了对推理能力与泛化性的准确评估。\n本文提出UTMath Benchmark（UTMath基准），这是一个稳健的评估框架，旨在通过大量单元测试评估大语言模型，同时关注模型回答的准确性与泛化性。该基准包含跨越9个数学领域的1053道前沿问题，每道题平均有68个测试用例。UTMath极具挑战性，表现最佳的模型o1-mini仅能解出32.57%的问题，其次是o1-preview，正确率为27.16%，GPT-4o以26.93%的正确率位列第三。\n此外，我们提出了Reasoning-to-Coding of Thoughts（RCoT，思维推理到编码）方法，该方法引导大语言模型在生成代码前进行显式推理，从而有助于生成更完善的解决方案，提升整体性能与效率。同时，我们还发布了UTMath-Train训练数据集（超过7万个样本），以支持学界进一步探索数学推理。可通过以下链接访问我们的基准：https://github.com/UTMathGroup/UTMath
英文摘要：
The evaluation of mathematical reasoning capabilities is essential for advancing Artificial General Intelligence (AGI). While Large Language Models (LLMs) have shown impressive performance in solving mathematical problems, existing benchmarks such as GSM8K and MATH present limitations, including narrow problem definitions with specific numbers and reliance on predetermined rules that hinder accurate assessments of reasoning and generality. This paper introduces the UTMath Benchmark, a robust evaluation framework designed to assess LLMs through extensive unit tests, with a focus on both the accuracy and generality of model responses. It comprises 1,053 cutting-edge problems spanning nine mathematical domains, with an average of 68 test cases per problem. UTMath is highly challenging, with the best-performing model, o1-mini, solving only 32.57\% of the problems, followed by o1-preview at 27.16\%, and GPT-4o at 26.93\%. Furthermore, we present the Reasoning-to-Coding of Thoughts (RCoT) approach, which encourages LLMs to engage in explicit reasoning prior to code generation, thereby facilitating the production of more sophisticated solutions and enhancing overall performance and efficiency. Additionally, we also release the UTMath-Train training dataset (more than 70k samples), to support the community in further exploring mathematical reasoning. Our benchmark can be accessed via the following link: https://github.com/UTMathGroup/UTMath
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
