arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2306.16636cs.CLcs.AIcs.LG
收藏

CMATH:你的语言模型能通过中国小学数学测试吗?

CMATH: Can Your Language Model Pass Chinese Elementary School Math Test?

Xiaomi AI Lab(小米人工智能实验室)

Tianwen Wei, Jian Luan, Wei Liu, Shuang Dong, Bin Wang

AI总结:

本文提出CMATH数据集,包含1.7k道中国小学数学应用题,用于评估LLMs的数学能力水平;实验发现仅GPT-4在所有六个年级中达到60%以上准确率且具备鲁棒性,其他模型表现不佳。

摘要:

我们提出了中国小学数学应用题(CMATH)数据集,包含1.7k道小学水平的数学应用题及详细标注,来源于真实的中国练习册和考试。该数据集旨在提供一个基准工具,用于评估以下问题:流行的大型语言模型(LLMs)的能力对应于小学数学的哪个年级水平?我们评估了多种流行的LLMs,包括商业和开源选项,发现只有GPT-4在所有六个小学年级中均取得成功(准确率≥60%),而其他模型在不同年级水平上表现不佳。此外,我们通过在CMATH数据集的原始题目中添加干扰信息,评估了几种表现最佳的LLMs的鲁棒性。研究结果表明,GPT-4能够保持鲁棒性,而其他模型则失败。我们预期本研究将揭示LLMs在算术和推理能力上的局限性,并促进其持续发展与进步。

英文摘要:

We present the Chinese Elementary School Math Word Problems (CMATH) dataset, comprising 1.7k elementary school-level math word problems with detailed annotations, source from actual Chinese workbooks and exams. This dataset aims to provide a benchmark tool for assessing the following question: to what grade level of elementary school math do the abilities of popular large language models (LLMs) correspond? We evaluate a variety of popular LLMs, including both commercial and open-source options, and discover that only GPT-4 achieves success (accuracy $\geq$ 60\%) across all six elementary school grades, while other models falter at different grade levels. Furthermore, we assess the robustness of several top-performing LLMs by augmenting the original problems in the CMATH dataset with distracting information. Our findings reveal that GPT-4 is able to maintains robustness, while other model fail. We anticipate that our study will expose limitations in LLMs' arithmetic and reasoning capabilities, and promote their ongoing development and advancement.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑