重新审视链式思维蒸馏中的容量差距：从实用角度出发
Revisiting the Capacity Gap in Chain-of-Thought Distillation from a Practical Perspective
AI总结：
本文从实用角度重新审视链式思维蒸馏中的容量差距问题，提出更现实的评估协议，发现容量差距影响并非在所有任务和设置中主导，尤其当候选教师性能差异显著时，为选择合适的教师-学生对提供实践指导。
摘要：
链式思维（CoT）蒸馏将推理行为从强教师转移到较小的学生，但先前工作报告了容量差距问题：当教师-学生能力不匹配较大时，蒸馏可能失败。我们从实用角度重新审视常用实验设置，发现CoT蒸馏通常比学生预蒸馏基线表现更差，这一问题在仅报告蒸馏后比较时被掩盖。因此，我们提出更现实的评估协议，并发现容量差距效应的影响并非在所有任务和设置中主导，尤其是在候选教师性能差异显著时。我们的结果为CoT蒸馏中选择教师-学生对提供了实践指导。
英文摘要：
Chain-of-thought (CoT) distillation transfers reasoning behaviors from a strong teacher to a smaller student, but prior work reports a capacity gap: distillation may fail when the teacher-student capability mismatch is large. We revisit the capacity gap from a practical perspective by re-examining commonly used experimental settings. Notably, we find that CoT distillation often degrades performance compared to the student's pre-distillation baseline, and that some settings used in prior work, while suitable for establishing the capacity gap as a phenomenon, do not reflect realistic deployment scenarios. Complementing prior work that establishes the capacity gap, we evaluate its practical impact under more realistic settings and find that it does not consistently dominate; stronger teachers tend to be preferable when candidate teachers differ substantially in performance. Our results offer practical guidance for selecting teacher-student pairs in CoT distillation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
