arXiv 2401.118642024-08-01cs.CLcs.AI

将数学推理能力蒸馏到小型语言模型中

Distilling Mathematical Reasoning Capabilities into Small Language Models

Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）

Xunyu Zhu, Jian Li, Yong Liu, Can Ma, Weiping Wang

备注：: Accepted for publication in Neural Networks

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出 EoTD 与 ETD，将方程及多种思维过程蒸馏进十亿参数以下 SLM，显著提升其数学推理并达到 SOTA。

摘要：

本工作致力于解决让先进大型语言模型（LLM）更普及的挑战，方法是在不损失性能的前提下，将其数学推理能力压缩到参数规模低于十亿的小型语言模型（SLM）中。我们提出 Equation-of-Thought Distillation（EoTD，方程思维蒸馏），这是一种新技术，将推理过程封装为基于方程的表示，从而构建用于微调 SLM 的 EoTD 数据集。此外，我们提出 Ensemble Thoughts Distillation（ETD，集成思维蒸馏）框架以提升 SLM 的推理性能。该框架会创建包含多种思维过程的推理数据集，其中包括 Chain-of-Thought（CoT，思维链）、Program-of-Thought（PoT，程序思维）和 Equation-of-Thought（EoT，方程思维），并使用该数据集进行微调。我们的实验性能表明，EoTD 显著提升了 SLM 的推理能力，而 ETD 使这些模型达到最先进的推理性能。

英文摘要：

This work addresses the challenge of democratizing advanced Large Language Models (LLMs) by compressing their mathematical reasoning capabilities into sub-billion parameter Small Language Models (SLMs) without compromising performance. We introduce Equation-of-Thought Distillation (EoTD), a novel technique that encapsulates the reasoning process into equation-based representations to construct an EoTD dataset for fine-tuning SLMs. Additionally, we propose the Ensemble Thoughts Distillation (ETD) framework to enhance the reasoning performance of SLMs. This involves creating a reasoning dataset with multiple thought processes, including Chain-of-Thought (CoT), Program-of-Thought (PoT), and Equation-of-Thought (EoT), and using it for fine-tuning. Our experimental performance demonstrates that EoTD significantly boosts the reasoning abilities of SLMs, while ETD enables these models to achieve state-of-the-art reasoning performance.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。