arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.17231cs.CL
收藏

MATHSENSEI:一种用于数学推理的工具增强型大语言模型

MATHSENSEI: A Tool-Augmented Large Language Model for Mathematical Reasoning

IIT Kharagpur(印度理工学院卡哈拉格普尔分校);Rakuten Group, Inc.(乐天集团);Rakuten Institute of Technology India(乐天印度技术研究院)

Debrup Das, Debopriyo Banerjee, Somak Aditya, Ashish Kulkarni

AI总结:

本文提出MathSensei,结合Bing、Python与Wolfram-Alpha等工具增强LLM数学推理,并通过MATH等数据集验证其在复杂问题上的显著收益。

摘要:

众所周知,工具增强型大语言模型(TALM)能够扩展大语言模型(LLM)的技能集,从而提升其在许多任务上的推理能力。尽管TALM已成功应用于不同的问答基准,但它们在复杂数学推理基准上的效果,以及知识检索工具和数学方程求解工具可能带来的互补收益,仍是尚待研究的问题。在本工作中,我们提出MathSensei,这是一种用于数学推理的工具增强型大语言模型。我们通过在数学推理数据集上的评估,研究了各类工具的互补收益,包括知识检索器(Bing Web Search)、程序生成器与执行器(Python),以及符号方程求解器(Wolfram-Alpha API)。我们在MATH这一常用于评估不同数学学科数学推理能力的流行数据集上进行了详尽的消融实验。我们还开展了涉及知名工具规划器的实验,以研究工具调用顺序对模型性能的影响。在MATH数据集上,MathSensei的准确率比采用思维链(Chain-of-Thought)的gpt-3.5-turbo高出13.5%。我们进一步观察到,TALM对于较简单的数学应用题(在GSM-8K中)效果并不显著,而随着问题复杂性和所需知识的增加,其收益也随之提升(在AQuA、MMLU-Math以及MATH中更高层次的复杂问题上逐步增加)。代码和数据可在https://github.com/Debrup-61/MathSensei获取。

英文摘要:

Tool-augmented Large Language Models (TALMs) are known to enhance the skillset of large language models (LLMs), thereby, leading to their improved reasoning abilities across many tasks. While, TALMs have been successfully employed in different question-answering benchmarks, their efficacy on complex mathematical reasoning benchmarks, and the potential complementary benefits offered by tools for knowledge retrieval and mathematical equation solving are open research questions. In this work, we present MathSensei, a tool-augmented large language model for mathematical reasoning. We study the complementary benefits of the tools - knowledge retriever (Bing Web Search), program generator + executor (Python), and symbolic equation solver (Wolfram-Alpha API) through evaluations on mathematical reasoning datasets. We perform exhaustive ablations on MATH, a popular dataset for evaluating mathematical reasoning on diverse mathematical disciplines. We also conduct experiments involving well-known tool planners to study the impact of tool sequencing on the model performance. MathSensei achieves 13.5% better accuracy over gpt-3.5-turbo with Chain-of-Thought on the MATH dataset. We further observe that TALMs are not as effective for simpler math word problems (in GSM-8K), and the benefit increases as the complexity and required knowledge increases (progressively over AQuA, MMLU-Math, and higher level complex questions in MATH). The code and data are available at https://github.com/Debrup-61/MathSensei.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑