PaD：程序辅助蒸馏比思维链微调更能有效教会小模型推理
PaD: Program-aided Distillation Can Teach Small Models Reasoning Better than Chain-of-thought Fine-tuning
AI总结：
本文提出程序辅助蒸馏（PaD），通过引入推理程序替代思维链并进行自动错误检查、错误注入和逐步束搜索，有效提升小模型的推理能力，使其在参数和数据规模更小的情况下超越部分大型模型及基线。
摘要：
尽管大型语言模型（LLMs）在各种自然语言处理任务中表现出色，但其庞大的规模和参数的不可访问性给实际部署带来了挑战。以往的研究尝试通过数据合成和思维链（CoT）微调，将特定任务的能力从LLMs蒸馏到较小的模型中。然而，合成的CoT数据通常包含错误的推理，这降低了蒸馏质量，尤其是在推理能力方面。在本研究中，我们提出了程序辅助蒸馏（PaD），该方法引入推理程序来抑制蒸馏数据中的错误，从而在推理任务上实现更好的蒸馏质量。在PaD中，我们利用推理程序替代CoT，允许对合成数据进行自动化错误检查。进一步地，通过错误注入和进一步训练，小型蒸馏模型能够迭代地自我完善推理过程。此外，我们通过逐步验证进行逐步束搜索，以获取更精确的推理链。我们在算术推理、符号推理和通用能力上评估了PaD。实验结果表明，使用PaD的小型模型不仅能优于某些LLMs（例如，LLaMA-1 13B），而且在参数和数据规模显著更小的情况下，也能在基线上实现大幅提升。源代码已在 https://github.com/Xuekai-Zhu/pad 上公开。
英文摘要：
While large language models (LLMs) excel in various natural language processing tasks, their huge size and the inaccessibility of parameters present challenges for practical deployment. Previous studies try to distill task-specific ability from LLMs to smaller models, using data synthesis and chain-of-thought (CoT) fine-tuning. However, synthetic CoT data often contains faulty reasoning, which deteriorates the quality of distillation, especially in reasoning capabilities. In this work, we propose Program-aided Distillation (PaD), which introduces reasoning programs to suppress the errors in distilled data, and thus achieves better distillation quality for reasoning tasks. In PaD, we utilize the reasoning program to substitute the CoT, allowing automated error checking of synthetic data. Further, through error injecting and further training, the small distilling model could iteratively self-refine the reasoning. Moreover, we conduct a step-wise beam search by step-by-step verifying to acquire more exact reasoning chains. We evaluate PaD on arithmetic reasoning, symbolic reasoning, and general ability. Experimental results demonstrate that smaller models using PaD can not only outperform certain LLMs~(e.g., LLaMA-1 13B) but also achieve strong improvement over baselines with a significantly smaller scale of parameters and data. The source code is publicly available at https://github.com/Xuekai-Zhu/pad.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
