arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.07747cs.CLcs.AI
收藏

FineMath:面向中文大型语言模型的细粒度数学评估基准

FineMath: A Fine-Grained Mathematical Evaluation Benchmark for Chinese Large Language Models

Tianjin University(天津大学);China University of Geosciences(中国地质大学);The University of Queensland(昆士兰大学)

Yan Liu, Renren Jin, Ling Shi, Zheng Yao, Deyi Xiong

AI总结:

本文提出 FineMath 细粒度中文数学评估基准,覆盖小学数学 17 类应用题并按推理步骤人工标注难度,实验揭示中文 LLM 数学推理仍有明显提升空间。

摘要:

为了全面评估大型语言模型(LLM)的数学推理能力,我们需要精心构建涵盖不同难度层次的多样数学概念与数学问题的评估数据集。为实现这一目标,本文提出 FineMath,这是一个用于评估中文 LLM 的细粒度数学评估基准数据集。FineMath 的构建覆盖了小学数学教授的主要关键数学概念,并进一步划分为 17 类数学应用题,从而能够深入分析 LLM 的数学推理能力。全部 17 类数学应用题均根据解题所需的推理步骤数量,人工标注了难度等级。我们在 FineMath 上对多种 LLM 进行了大量实验,发现中文 LLM 在数学推理能力方面仍有相当大的提升空间。我们还对以往被忽视的评估流程和评估方法进行了深入分析;这两个因素会显著影响模型结果以及我们对其数学推理能力的理解。该数据集将很快公开发布。

英文摘要:

To thoroughly assess the mathematical reasoning abilities of Large Language Models (LLMs), we need to carefully curate evaluation datasets covering diverse mathematical concepts and mathematical problems at different difficulty levels. In pursuit of this objective, we propose FineMath in this paper, a fine-grained mathematical evaluation benchmark dataset for assessing Chinese LLMs. FineMath is created to cover the major key mathematical concepts taught in elementary school math, which are further divided into 17 categories of math word problems, enabling in-depth analysis of mathematical reasoning abilities of LLMs. All the 17 categories of math word problems are manually annotated with their difficulty levels according to the number of reasoning steps required to solve these problems. We conduct extensive experiments on a wide range of LLMs on FineMath and find that there is still considerable room for improvements in terms of mathematical reasoning capability of Chinese LLMs. We also carry out an in-depth analysis on the evaluation process and methods that have been overlooked previously. These two factors significantly influence the model results and our understanding of their mathematical reasoning capabilities. The dataset will be publicly available soon.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑