超越模仿：在推理蒸馏中从双重思维链学习关键推理步骤
Beyond Imitation: Learning Key Reasoning Steps from Dual Chain-of-Thoughts in Reasoning Distillation
AI总结：
针对SLMs仅模仿教师CoT形式而难以学习关键推理步骤的问题，提出EDIT方法，通过生成结论分歧的双重CoTs并利用最小编辑距离定位关键步骤进行优化，显著提升了推理蒸馏效果。
摘要：
随着大型语言模型（LLMs）规模扩大并获得强大的思维链推理能力，实际的资源限制促使人们努力将这些能力蒸馏到更紧凑的小型语言模型中。我们发现CoTs主要由简单的推理形式组成，只有一小部分（约4.7%）的关键推理步骤真正影响结论。然而，以往的蒸馏方法通常仅对教师LLMs生成的正确CoTs数据进行监督微调，导致学生模型难以学习关键推理步骤，而是模仿教师的推理形式，在这些步骤上犯错或遗漏。为解决这些问题，借鉴人类学习中根据正确方案分析错误往往能揭示成败关键步骤的类比，我们提出错误驱动的关键推理步骤蒸馏（EDIT），这是一种进一步帮助SLMs学习关键推理步骤而非仅进行简单微调的新方法。首先，为暴露CoTs中的关键步骤，我们设计特定提示生成具有相似推理路径但结论分歧的双重CoTs数据。然后，在双重CoTs数据上应用最小编辑距离算法定位这些关键步骤，并优化这些步骤的似然性。大量实验验证了EDIT在域内和域外基准推理数据集上的有效性。进一步分析表明，EDIT能生成包含更多正确关键推理步骤的高质量CoTs。值得注意的是，我们还探索了不同错误模式如何影响性能，发现与双重CoTs中的知识或数学计算错误相比，EDIT从逻辑错误中获益更多。
英文摘要：
As Large Language Models (LLMs) scale up and gain powerful Chain-of-Thoughts (CoTs) reasoning abilities, practical resource constraints drive efforts to distill these capabilities into more compact Smaller Language Models (SLMs). We find that CoTs consist mainly of simple reasoning forms, with a small proportion ($\approx 4.7\%$) of key reasoning steps that truly impact conclusions. However, previous distillation methods typically involve supervised fine-tuning student SLMs only on correct CoTs data produced by teacher LLMs, resulting in students struggling to learn the key reasoning steps, instead imitating the teacher's reasoning forms and making errors or omissions on these steps. To address these issues, drawing an analogy to human learning, where analyzing mistakes according to correct solutions often reveals the crucial steps leading to successes or failures, we propose mistak\textbf{E}-\textbf{D}riven key reason\textbf{I}ng step distilla\textbf{T}ion (\textbf{EDIT}), a novel method that further aids SLMs learning key reasoning steps rather than mere simple fine-tuning. Firstly, to expose these crucial steps in CoTs, we design specific prompts to generate dual CoTs data with similar reasoning paths but divergent conclusions. Then, we apply the minimum edit distance algorithm on the dual CoTs data to locate these key steps and optimize the likelihood of these steps. Extensive experiments validate the effectiveness of EDIT across both in-domain and out-of-domain benchmark reasoning datasets. Further analysis shows that EDIT can generate high-quality CoTs with more correct key reasoning steps. Notably, we also explore how different mistake patterns affect performance and find that EDIT benefits more from logical errors than from knowledge or mathematical calculation errors in dual CoTs\footnote{Code can be found at \url{https://github.com/C-W-D/EDIT}}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
