基于关键点的渐进式思维链蒸馏方法用于大型语言模型
Keypoint-based Progressive Chain-of-Thought Distillation for LLMs
AI总结：
提出KPOD框架，通过掩码学习加权关键点token和渐进式蒸馏策略，解决思维链蒸馏中关键点模仿不足和步骤学习顺序问题，在四个推理基准上显著超越现有方法。
摘要：
思维链蒸馏是一种强大的技术，用于将大型语言模型（LLMs）的推理能力迁移到较小的学生模型。以往的方法通常要求学生模型模仿LLMs生成的逐步推理过程，但往往面临以下挑战：（i）推理过程中的各个token重要性不同，若同等对待可能无法准确模仿关键点token，导致推理错误。（ii）它们通常通过一致地预测推理过程中的所有步骤来蒸馏知识，这未能区分步骤生成的学习顺序。这与人类从易到难的认知发展过程相悖，导致次优结果。为此，我们提出了一个统一框架，称为KPOD，以解决这些问题。具体来说，我们提出了一种利用掩码学习的token加权模块，以鼓励学生在蒸馏过程中准确模仿关键点token。此外，我们开发了一种推理过程内的渐进式蒸馏策略，首先训练学生生成最终推理步骤，然后逐步扩展到覆盖整个推理过程。为实现这一目标，我们提出了一种加权token生成损失来评估步骤推理难度，并设计了一个价值函数，综合考虑步骤难度和问题多样性来调度渐进式蒸馏。在四个推理基准上的大量实验表明，我们的KPOD大幅优于以往方法。
英文摘要：
Chain-of-thought distillation is a powerful technique for transferring reasoning abilities from large language models (LLMs) to smaller student models. Previous methods typically require the student to mimic the step-by-step rationale produced by LLMs, often facing the following challenges: (i) Tokens within a rationale vary in significance, and treating them equally may fail to accurately mimic keypoint tokens, leading to reasoning errors. (ii) They usually distill knowledge by consistently predicting all the steps in a rationale, which falls short in distinguishing the learning order of step generation. This diverges from the human cognitive progression of starting with easy tasks and advancing to harder ones, resulting in sub-optimal outcomes. To this end, we propose a unified framework, called KPOD, to address these issues. Specifically, we propose a token weighting module utilizing mask learning to encourage accurate mimicry of keypoint tokens by the student during distillation. Besides, we develop an in-rationale progressive distillation strategy, starting with training the student to generate the final reasoning steps and gradually extending to cover the entire rationale. To accomplish this, a weighted token generation loss is proposed to assess step reasoning difficulty, and a value function is devised to schedule the progressive distillation by considering both step difficulty and question diversity. Extensive experiments on four reasoning benchmarks illustrate our KPOD outperforms previous methods by a large margin.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
