arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.08196cs.CLcs.AIcs.CV
收藏

MathCoder2:基于模型翻译的数学代码进行持续预训练以获得更好的数学推理能力

MathCoder2: Better Math Reasoning from Continued Pretraining on Model-translated Mathematical Code

Multimedia Laboratory (MMLab), The Chinese University of Hong Kong(香港中文大学多媒体实验室(MMLab))

Zimu Lu, Aojun Zhou, Ke Wang, Houxing Ren, Weikang Shi, Junting Pan, Mingjie Zhan, Hongsheng Li

AI总结:

提出生成附带推理步骤的数学代码方法,构建19.2B token的MathCode-Pile语料库进行持续预训练,显著提升模型数学推理能力,并开源全部代码。

摘要:

由于代码的精确性和准确性,它已被证明能有效增强大型语言模型的数学推理能力。以往涉及数学持续预训练的工作通常包含使用数学相关包的代码,但这些包主要为工程、机器学习、信号处理或模块测试等领域设计,而非直接聚焦于数学推理。本文提出了一种新方法,用于生成附带相应推理步骤的数学代码以进行持续预训练。我们的方法首先构建高质量的数学持续预训练数据集,整合了数学相关网络数据、使用数学包的代码、数学教科书和合成数据。接着,我们从已收集的数据集中提取LaTeX表达式、表达式所需条件及表达式结果来构建推理步骤。基于提取的信息,我们生成相应代码以准确捕捉数学推理过程。将生成的代码附加到每个推理步骤后,得到由自然语言推理步骤与对应代码配对组成的数据。将此数据与原始数据集结合,构建了包含19.2B token的高性能数学预训练语料库MathCode-Pile。使用该语料库训练多个流行基础模型显著提升了其数学能力,由此诞生了MathCoder2系列模型。所有数据处理和训练代码均已开源,确保整个数据收集和训练流程的完全透明与易于复现。代码发布于https://github.com/mathllm/MathCoder2。

英文摘要:

Code has been shown to be effective in enhancing the mathematical reasoning abilities of large language models due to its precision and accuracy. Previous works involving continued mathematical pretraining often include code that utilizes math-related packages, which are primarily designed for fields such as engineering, machine learning, signal processing, or module testing, rather than being directly focused on mathematical reasoning. In this paper, we introduce a novel method for generating mathematical code accompanied with corresponding reasoning steps for continued pretraining. Our approach begins with the construction of a high-quality mathematical continued pretraining dataset by incorporating math-related web data, code using mathematical packages, math textbooks, and synthetic data. Next, we construct reasoning steps by extracting LaTeX expressions, the conditions needed for the expressions, and the results of the expressions from the previously collected dataset. Based on this extracted information, we generate corresponding code to accurately capture the mathematical reasoning process. Appending the generated code to each reasoning step results in data consisting of paired natural language reasoning steps and their corresponding code. Combining this data with the original dataset results in a 19.2B-token high-performing mathematical pretraining corpus, which we name MathCode-Pile. Training several popular base models with this corpus significantly improves their mathematical abilities, leading to the creation of the MathCoder2 family of models. All of our data processing and training code is open-sourced, ensuring full transparency and easy reproducibility of the entire data collection and training pipeline. The code is released at https://github.com/mathllm/MathCoder2 .

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑