arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.02884cs.CLcs.AIcs.LG
收藏

MathScale: 面向数学推理的指令调优规模扩展

MathScale: Scaling Instruction Tuning for Mathematical Reasoning

The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Microsoft Research Asia(微软亚洲研究院);Shenzhen Research Institute of Big Data(深圳大数据研究院)

Zhengyang Tang, Xingxing Zhang, Benyou Wang, Furu Wei

AI总结:

针对大语言模型数学推理能力不足的问题,提出可扩展方法MathScale,基于概念图生成200万道数学题数据集MathScaleQA,构建多难度基准MwpBench,微调后同规模模型数学推理性能达SOTA。

摘要:

大语言模型(LLM)已展现出卓越的问题求解能力,但在数学问题求解方面的水平仍有不足。我们提出MathScale,这是一种利用前沿LLM(例如GPT-3.5)生成高质量数学推理数据的简单且可扩展的方法。受人类数学学习中的认知机制启发,该方法首先从种子数学题中提取主题与知识点,构建概念图,再利用该概念图生成新的数学题。MathScale在我们生成的数学数据集规模维度上展现出有效的可扩展性,最终我们构建了包含200万道数学问答对的数学推理数据集MathScaleQA。为全面评估LLM的数学推理能力,我们构建了数学应用题基准MwpBench,它整合了10个数据集(含GSM8K和MATH),覆盖K-12、大学及竞赛级别的数学问题。我们使用MathScaleQA微调开源LLM(例如LLaMA-2和Mistral),其数学推理能力得到显著提升。在MwpBench上评估显示,MathScale-7B在所有数据集上均取得了当前最优性能,微平均准确率和宏平均准确率分别比同规模的最优同类模型高出42.9%和43.7%。

英文摘要:

Large language models (LLMs) have demonstrated remarkable capabilities in problem-solving. However, their proficiency in solving mathematical problems remains inadequate. We propose MathScale, a simple and scalable method to create high-quality mathematical reasoning data using frontier LLMs (e.g., {\tt GPT-3.5}). Inspired by the cognitive mechanism in human mathematical learning, it first extracts topics and knowledge points from seed math questions and then build a concept graph, which is subsequently used to generate new math questions. MathScale exhibits effective scalability along the size axis of the math dataset that we generate. As a result, we create a mathematical reasoning dataset (MathScaleQA) containing two million math question-answer pairs. To evaluate mathematical reasoning abilities of LLMs comprehensively, we construct {\sc MwpBench}, a benchmark of Math Word Problems, which is a collection of ten datasets (including GSM8K and MATH) covering K-12, college, and competition level math problems. We apply MathScaleQA to fine-tune open-source LLMs (e.g., LLaMA-2 and Mistral), resulting in significantly improved capabilities in mathematical reasoning. Evaluated on {\sc MwpBench}, MathScale-7B achieves state-of-the-art performance across all datasets, surpassing its best peers of equivalent size by 42.9\% in micro average accuracy and 43.7\% in macro average accuracy, respectively.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑