IGC:将门控计算器集成到LLM中以可靠且高效地解决算术任务
IGC: Integrating a Gated Calculator into an LLM to Solve Arithmetic Tasks Reliably and Efficiently
AI总结:
本文提出集成门控计算器(IGC)模块,使LLM在GPU上模拟计算器执行算术,微调Llama后在BigBench Arithmetic上以98%-99%准确率超越所有模型,并首次解决乘法子任务。
摘要:
解决算术任务是一项简单而基础的技能,然而现代大型语言模型(LLMs)在这方面存在很大困难。我们引入了集成门控计算器(IGC),这是一个使LLM能够通过在GPU上模拟计算器来执行算术运算的模块。我们使用该模块对Llama模型进行微调,并在BigBench Arithmetic基准上对其进行测试,在该基准上它超越了当前最优水平,优于基准上的所有模型,包括规模大近两个数量级的模型。我们的方法只需一次迭代即可运行,并且不需要外部工具。它完全在LLM内部执行算术运算,无需生成中间token。该方法计算高效、可解释,并且对不需要算术运算的任务避免了副作用。在多次训练运行中以及对于所有子任务,它都能可靠地达到98%到99%的准确率,包括此前尚未解决的、难度显著更高的乘法子任务。
英文摘要:
Solving arithmetic tasks is a simple and fundamental skill, yet modern Large Language Models (LLMs) have great difficulty with them. We introduce the Integrated Gated Calculator (IGC), a module that enables LLMs to perform arithmetic by emulating a calculator on the GPU. We finetune a Llama model with our module and test it on the BigBench Arithmetic benchmark, where it beats the State of the Art, outperforming all models on the benchmark, including models almost two orders of magnitude larger. Our approach takes only a single iteration to run and requires no external tools. It performs arithmetic operations entirely inside the LLM without the need to produce intermediate tokens. It is computationally efficient, interpretable, and avoids side-effects on tasks that do not require arithmetic operations. It reliably achieves 98\% to 99\% accuracy across multiple training runs and for all subtasks, including the substantially harder subtask of multiplication, which was previously unsolved.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。