arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.11451cs.CLcs.AI
收藏

SciAgent:面向科学推理的工具增强语言模型

SciAgent: Tool-augmented Language Models for Scientific Reasoning

Microsoft(微软公司);University of California, Santa Barbara(加利福尼亚大学圣巴巴拉分校);Tsinghua University(清华大学);Singapore Management University(新加坡管理大学);Nanyang Technological University(南洋理工大学)

Yubo Ma, Zhibin Gou, Junheng Hao, Ruochen Xu, Shuohang Wang, Liangming Pan, Yujiu Yang, Yixin Cao, Aixin Sun, Hany Awadalla, Weizhu Chen

AI总结:

本文提出工具增强科学推理任务,构建含3万样本和6千工具的MathFunc语料库,开发SciAgent模型及SciToolBench基准,实验表明其显著优于同规模模型和ChatGPT。

摘要:

科学推理即使对最先进的大语言模型(LLMs)也构成了极大的挑战。为了使这一任务对LLMs更具实用性且更可解,我们引入了一种名为工具增强科学推理的新任务设置。该设置为LLMs补充了可扩展的工具集,并将重点从追求一个无所不知的问题求解器转向一个熟练的工具使用者。为了促进该设置的研究,我们构建了一个名为MathFunc的工具增强训练语料库,其中包含超过30,000个样本和大约6,000个工具。基于MathFunc,我们开发了SciAgent,用于检索、理解并在必要时使用工具来解决科学问题。此外,我们构建了一个名为SciToolBench的基准,涵盖五个科学领域,以评估LLMs在工具辅助下的能力。在SciToolBench上的大量实验证实了SciAgent的有效性。值得注意的是,SciAgent-Mistral-7B在绝对准确率上超过同规模的其他LLMs超过13%。此外,SciAgent-DeepMath-7B表现出远优于ChatGPT的性能。

英文摘要:

Scientific reasoning poses an excessive challenge for even the most advanced Large Language Models (LLMs). To make this task more practical and solvable for LLMs, we introduce a new task setting named tool-augmented scientific reasoning. This setting supplements LLMs with scalable toolsets, and shifts the focus from pursuing an omniscient problem solver to a proficient tool-user. To facilitate the research of such setting, we construct a tool-augmented training corpus named MathFunc which encompasses over 30,000 samples and roughly 6,000 tools. Building on MathFunc, we develop SciAgent to retrieve, understand and, if necessary, use tools for scientific problem solving. Additionally, we craft a benchmark, SciToolBench, spanning five scientific domains to evaluate LLMs' abilities with tool assistance. Extensive experiments on SciToolBench confirm the effectiveness of SciAgent. Notably, SciAgent-Mistral-7B surpasses other LLMs with the same size by more than 13% in absolute accuracy. Furthermore, SciAgent-DeepMath-7B shows much superior performance than ChatGPT.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑