arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.02893cs.CL
收藏

ChatGLM-Math:通过自我批评流水线提升大型语言模型的数学问题求解能力

ChatGLM-Math: Improving Math Problem-Solving in Large Language Models with a Self-Critique Pipeline

Tsinghua University(清华大学)

Yifan Xu, Xiao Liu, Xinghan Liu, Zhenyu Hou, Yueyan Li, Xiaohan Zhang, Zihan Wang, Aohan Zeng, Zhengxiao Du, Wenyi Zhao, Jie Tang, Yuxiao Dong

AI总结:

本文提出Self-Critique流水线,通过Math-Critique模型、拒绝式微调和DPO在对齐阶段同步提升ChatGLM3-32B的数学与语言能力。

摘要:

大型语言模型(LLM)已展现出对人类语言的出色掌握,但在需要数学问题求解的真实应用中仍显吃力。尽管已有许多增强LLM数学能力的策略和数据集被开发出来,但在已部署的LLM系统中同时保持并提升语言能力与数学能力仍是一项挑战。在本工作中,我们定制了Self-Critique(自我批评)流水线,在LLM对齐的反馈学习阶段应对该挑战。我们首先从LLM自身训练一个通用的Math-Critique模型以提供反馈信号。随后,我们在LLM自身生成的内容上依次采用拒绝式微调和直接偏好优化来收集数据。基于ChatGLM3-32B,我们在学术数据集以及我们新构建的具有挑战性的数据集MathUserEval上开展了一系列实验。结果表明,我们的流水线显著增强了LLM的数学问题求解能力,同时仍提升了其语言能力,表现优于可能大两倍的LLM。相关技术已部署到在线服务LLM ChatGLM(https://chatglm.cn)。相关评估数据集和脚本发布于https://github.com/THUDM/ChatGLM-Math。

英文摘要:

Large language models (LLMs) have shown excellent mastering of human language, but still struggle in real-world applications that require mathematical problem-solving. While many strategies and datasets to enhance LLMs' mathematics are developed, it remains a challenge to simultaneously maintain and improve both language and mathematical capabilities in deployed LLM systems.In this work, we tailor the Self-Critique pipeline, which addresses the challenge in the feedback learning stage of LLM alignment. We first train a general Math-Critique model from the LLM itself to provide feedback signals. Then, we sequentially employ rejective fine-tuning and direct preference optimization over the LLM's own generations for data collection. Based on ChatGLM3-32B, we conduct a series of experiments on both academic and our newly created challenging dataset, MathUserEval. Results show that our pipeline significantly enhances the LLM's mathematical problem-solving while still improving its language ability, outperforming LLMs that could be two times larger. Related techniques have been deployed to ChatGLM\footnote{\url{https://chatglm.cn}}, an online serving LLM. Related evaluation dataset and scripts are released at \url{https://github.com/THUDM/ChatGLM-Math}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑