揭示思维链推理蒸馏的关键因素
Unveiling the Key Factors for Distilling Chain-of-Thought Reasoning
AI总结：
本研究系统考察了影响思维链蒸馏的关键因素，发现粒度、格式和教师模型对小型语言模型的影响与大型语言模型不同，强调需针对具体学生模型定制蒸馏策略。
摘要：
大型语言模型（LLMs）通过思维链（CoT）提示在推理任务中表现出色。然而，CoT提示极大地增加了计算需求，这促使人们越来越关注将CoT能力蒸馏到小型语言模型（SLMs）中。本研究系统地考察了影响CoT蒸馏的因素，包括粒度、格式和教师模型的选择。通过涉及四个教师模型和七个学生模型在七个数学和常识推理数据集上的实验，我们发现了三个关键发现：（1）与LLMs不同，SLMs与粒度之间呈现非单调关系，更强的模型受益于更细粒度的推理，而较弱的模型在更简单的CoT监督下表现更好；（2）CoT格式对LLMs影响显著，但对SLMs影响甚微，这可能是由于它们依赖监督微调而非预训练偏好；（3）更强的教师模型并不总是产生更好的学生模型，因为CoT监督的多样性和复杂性可能超过准确性本身。这些发现强调需要针对特定学生模型定制CoT策略，为优化SLMs中的CoT蒸馏提供了可操作的见解。代码和数据集可在https://github.com/EIT-NLP/Distilling-CoT-Reasoning获取。
英文摘要：
Large Language Models (LLMs) excel in reasoning tasks through Chain-of-Thought (CoT) prompting. However, CoT prompting greatly increases computational demands, which has prompted growing interest in distilling CoT capabilities into Small Language Models (SLMs). This study systematically examines the factors influencing CoT distillation, including the choice of granularity, format and teacher model. Through experiments involving four teacher models and seven student models across seven mathematical and commonsense reasoning datasets, we uncover three key findings: (1) Unlike LLMs, SLMs exhibit a non-monotonic relationship with granularity, with stronger models benefiting from finer-grained reasoning and weaker models performing better with simpler CoT supervision; (2) CoT format significantly impacts LLMs but has minimal effect on SLMs, likely due to their reliance on supervised fine-tuning rather than pretraining preferences; (3) Stronger teacher models do NOT always produce better student models, as diversity and complexity in CoT supervision can outweigh accuracy alone. These findings emphasize the need to tailor CoT strategies to specific student model, offering actionable insights for optimizing CoT distillation in SLMs. The code and datasets are available at https://github.com/EIT-NLP/Distilling-CoT-Reasoning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
