arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2306.02408cs.CL
收藏

评估与改进工具增强的计算密集型数学推理

Evaluating and Improving Tool-Augmented Computation-Intensive Math Reasoning

Sichuan University(四川大学);iFLYTEK Research(讯飞研究院);Renmin University of China(中国人民大学);iFLYTEK AI Research (Central China)(讯飞人工智能研究院(华中))

Beichen Zhang, Kun Zhou, Xilin Wei, Wayne Xin Zhao, Jing Sha, Shijin Wang, Ji-Rong Wen

AI总结:

针对工具增强数学推理缺乏中间步骤评估的问题,本文构建含4,886题的中文CARP数据集,并提出通过工具接口反复审校推理步骤的DELI方法,显著提升性能。

摘要:

思维链提示(Chain-of-thought prompting,CoT)和工具增强已在近期研究中被验证为提升大型语言模型(LLM)在复杂数学相关任务上执行逐步推理能力的有效实践。然而,大多数现有数学推理数据集可能无法充分评估和分析 LLM 操作工具并执行推理的能力,因为这些数据集可能只需要极少的工具调用,或缺少用于评估中间推理步骤的标注。为解决该问题,我们构建了 CARP,这是一个新的中文数据集,包含 4,886 道计算密集型代数题,并对中间步骤进行了规范化标注。在 CARP 中,我们测试了四个采用 CoT 提示的 LLM,发现它们都容易在求解的早期步骤犯错,从而导致答案错误。基于这一发现,我们提出了一种能够借助工具接口对推理步骤进行审慎推演的新方法,即 DELI。在 DELI 中,我们首先基于检索到的示例初始化一个逐步解答,然后迭代两个审慎推演过程,分别从工具操作和自然语言推理的角度检查并完善所生成解答的中间步骤,直到获得收敛的解答或达到最大轮数。在 CARP 和另外六个数据集上的实验结果表明,所提出的 DELI 大多优于具有竞争力的基线,并能够进一步提升现有 CoT 方法的性能。我们的数据和代码可在 https://github.com/RUCAIBox/CARP 获取。

英文摘要:

Chain-of-thought prompting~(CoT) and tool augmentation have been validated in recent work as effective practices for improving large language models~(LLMs) to perform step-by-step reasoning on complex math-related tasks. However, most existing math reasoning datasets may be not able to fully evaluate and analyze the ability of LLMs in manipulating tools and performing reasoning, as they may only require very few invocations of tools or miss annotations for evaluating intermediate reasoning steps. To address the issue, we construct \textbf{CARP}, a new Chinese dataset consisting of 4,886 computation-intensive algebra problems with formulated annotations on intermediate steps. In CARP, we test four LLMs with CoT prompting, and find that they are all prone to make mistakes at the early steps of the solution, leading to wrong answers. Based on this finding, we propose a new approach that can deliberate the reasoning steps with tool interfaces, namely \textbf{DELI}. In DELI, we first initialize a step-by-step solution based on retrieved exemplars, then iterate two deliberation procedures that check and refine the intermediate steps of the generated solution, from the perspectives of tool manipulation and natural language reasoning, until obtaining converged solutions or reaching the maximum turn. Experimental results on CARP and six other datasets show that the proposed DELI mostly outperforms competitive baselines, and can further boost the performance of existing CoT methods. Our data and code are available in \url{https://github.com/RUCAIBox/CARP}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑