众所周知,工具增强型大语言模型(TALM)能够扩展大语言模型(LLM)的技能集,从而提升其在许多任务上的推理能力。尽管TALM已成功应用于不同的问答基准,但它们在复杂数学推理基准上的效果,以及知识检索工具和数学方程求解工具可能带来的互补收益,仍是尚待研究的问题。在本工作中,我们提出MathSensei,这是一种用于数学推理的工具增强型大语言模型。我们通过在数学推理数据集上的评估,研究了各类工具的互补收益,包括知识检索器(Bing Web Search)、程序生成器与执行器(Python),以及符号方程求解器(Wolfram-Alpha API)。我们在MATH这一常用于评估不同数学学科数学推理能力的流行数据集上进行了详尽的消融实验。我们还开展了涉及知名工具规划器的实验,以研究工具调用顺序对模型性能的影响。在MATH数据集上,MathSensei的准确率比采用思维链(Chain-of-Thought)的gpt-3.5-turbo高出13.5%。我们进一步观察到,TALM对于较简单的数学应用题(在GSM-8K中)效果并不显著,而随着问题复杂性和所需知识的增加,其收益也随之提升(在AQuA、MMLU-Math以及MATH中更高层次的复杂问题上逐步增加)。代码和数据可在https://github.com/Debrup-61/MathSensei获取。
英文摘要:
Tool-augmented Large Language Models (TALMs) are known to enhance the skillset of large language models (LLMs), thereby, leading to their improved reasoning abilities across many tasks. While, TALMs have been successfully employed in different question-answering benchmarks, their efficacy on complex mathematical reasoning benchmarks, and the potential complementary benefits offered by tools for knowledge retrieval and mathematical equation solving are open research questions. In this work, we present MathSensei, a tool-augmented large language model for mathematical reasoning. We study the complementary benefits of the tools - knowledge retriever (Bing Web Search), program generator + executor (Python), and symbolic equation solver (Wolfram-Alpha API) through evaluations on mathematical reasoning datasets. We perform exhaustive ablations on MATH, a popular dataset for evaluating mathematical reasoning on diverse mathematical disciplines. We also conduct experiments involving well-known tool planners to study the impact of tool sequencing on the model performance. MathSensei achieves 13.5% better accuracy over gpt-3.5-turbo with Chain-of-Thought on the MATH dataset. We further observe that TALMs are not as effective for simpler math word problems (in GSM-8K), and the benefit increases as the complexity and required knowledge increases (progressively over AQuA, MMLU-Math, and higher level complex questions in MATH). The code and data are available at https://github.com/Debrup-61/MathSensei.
School of Computer Science and Technology, East China Normal University(华东师范大学计算机科学与技术学院);Lab of Artificial Intelligence for Education, East China Normal University(华东师范大学教育人工智能实验室)
Enhancing Test-Time Scaling of Large Language Models with Hierarchical Retrieval-Augmented MCTS
The Ohio State University(俄亥俄州立大学);Case Western Reserve University(凯斯西储大学);Duke University(杜克大学);University of Hong Kong(香港大学);City University of Hong Kong(香港城市大学);ByteDance(字节跳动)
Consistent Autoformalization for Constructing Mathematical Libraries
Department of Computer Science, University of Manchester(曼彻斯特大学计算机科学系);Idiap Research Institute(Idiap研究 institute);National Biomarker Centre, CRUK Manchester Institute(国家生物标志物中心,CRUK曼彻斯特研究所)
The Periodic Table of LLM Reasoning: A Structured Survey of Reasoning Paradigms, Methods, and Failure Modes
Singapore Institute of Technology(新加坡理工大学);Nvidia AI Center (SNAIC)(英伟达人工智能中心(SNAIC));MIDAS Lab, IIIT Delhi(IIIT德里MIDAS实验室);MIDAS Lab, IIT Mandi(IIT曼迪MIDAS实验室);Owl Autonomous Imaging, Inc.(Owl自主成像公司);College of Computing & Data Science, NTU Singapore(新加坡南洋理工大学计算与数据科学学院);NVIDIA AI Technology Centre, Singapore(英伟达新加坡人工智能技术中心);Department of Computer Science and Engineering, IIT Kanpur(IIT坎普尔计算机科学与工程系)
FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios
Shanghai Jiao Tong University(上海交通大学);Carnegie Mellon University(卡内基梅隆大学);City University of Hong Kong(香港城市大学);New York University(纽约大学);Meta AI;The Hong Kong University of Science and Technology(香港科技大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
Harder Is Better: Boosting Mathematical Reasoning via Difficulty-Aware GRPO and Multi-Aspect Question Reformulation
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学人工智能学院);AMAP, Alibaba Group(阿里集团AMAP);Xiamen University(厦门大学);Dalian University of Technology(大连理工大学)
FG-PRM: Fine-grained Hallucination Detection and Mitigation in Language Model Mathematical Reasoning
Department of Computer Science, University of Texas at Dallas(德克萨斯大学达拉斯分校计算机科学系);Department of Electrical Engineering Computer Science, University of Michigen(密歇根大学电子工程与计算机科学系)