啄木鸟蒸馏：弱模型诊断强模型中的推理错误

中文摘要 AI 辅助

大型语言模型尽管具备解决推理任务的能力，却常常在这类任务上失败。我们认为，这类失败多源于中间步骤的局部推理错误，而非全局能力不足。我们表明，这些错误通常可修复：在强模型推理前缀后插入由弱探测模型生成的简短补丁，可将推理轨迹导向正确解决方案。然而，直接在弱补丁或修复后的轨迹上微调，并不能可靠地内化这种校正效果，这表明有用信号不在于干预文本本身，而在于它如何重塑模型未来的推理分布。因此，我们提出Woodpecker Distillation，这是一种弱到强的训练框架，从对比局部干预中学习。我们的方法对比同一前缀下成功与不成功的弱模型补丁，从它们诱导的未来 token 预测中构建校正教师分布，并将该信号蒸馏到强模型中。在数学推理基准上的实验表明，Woodpecker Distillation 持续提升强模型性能，且优于直接模仿基线。

英文摘要

Large language models often fail on reasoning tasks despite possessing the capability to solve them. We argue that many such failures arise from localized reasoning bugs in intermediate steps rather than from global incompetence. We show that these bugs are frequently repairable: inserting a short patch generated by a weak probe model after the same strong-model reasoning prefix can redirect the trajectory toward a correct solution. However, this corrective effect is not reliably internalized by directly fine-tuning on weak patches or repaired trajectories, suggesting that the useful signal lies not in the intervention text itself, but in how it reshapes the model's future reasoning distribution. We therefore propose Woodpecker Distillation, a weak-to-strong training framework that learns from contrastive local interventions. Our method contrasts successful and unsuccessful weak-model patches at the same prefix, constructs a corrective teacher distribution from their induced future token predictions, and distills this signal into the strong model. Experiments on mathematical reasoning benchmarks show that Woodpecker Distillation consistently improves strong-model performance and outperforms direct imitation baselines.

发表机构

• Baidu Inc.（百度公司）
• Nanyang Technological University（南洋理工大学）

机构由 AI 辅助整理，请以论文原文为准。