揭示小规模微调在R1风格强化学习中的有效性
Towards Revealing the Effectiveness of Small-Scale Fine-tuning in R1-style Reinforcement Learning
AI总结：
本研究提出重蒸馏技术，通过从RL训练策略采样提升小规模SFT效率，以少量样本匹配RL性能，并揭示R1风格RL机制。
摘要：
R1风格强化学习（RL）显著增强了大型语言模型的推理能力，然而基于规则的RL背后的机制仍不清楚。我们发现小规模监督微调（SFT）对RL有显著影响，但效率较低。为了解释我们的观察，我们提出了一个分析框架，并通过测量样本效应来比较SFT和RL的效率。我们的假设性分析显示了提高SFT效率的潜力。在我们的分析指导下，我们提出了重蒸馏（Re-distillation），这是一种通过从RL训练的策略中采样来提升小规模蒸馏效果的技术。重蒸馏在三个数据集以及Qwen和Llama模型上均表现出惊人且一致的效率：重蒸馏模型以远少于RL的样本和计算量达到了RL的性能。结果，在K&K数据集上，我们重蒸馏的Qwen-2.5-1.5B模型仅用1K SFT样本就超越了DeepSeek-V3-0324。我们证明了重蒸馏可用于高效平衡RL中的多个目标。我们的工作解释了R1风格RL中的几个有趣现象，揭示了其经验成功背后的机制。代码可在以下网址获取：https://github.com/on1262/deep-reasoning。
英文摘要：
R1-style Reinforcement Learning (RL) significantly enhances Large Language Models' reasoning capabilities, yet the mechanism behind rule-based RL remains unclear. We found that small-scale SFT has substantial influence on RL but shows poor efficiency. To explain our observations, we propose an analytical framework and compare the efficiency of SFT and RL by measuring \textbf{sample effect}. Our hypothetical analysis shows the potential to improve SFT efficiency. Guided by our analysis, we propose \textbf{Re-distillation}, a technique that aims to boost the effectiveness of small-scale distillation by sampling from the RL-trained policy. Re-distillation shows consistent surprising efficiency on three datasets and both Qwen\&Llama models: Re-distilled models matched RL performance with far fewer samples and less computation. As a result, on K\&K dataset, our re-distilled Qwen-2.5-1.5B model surpasses DeepSeek-V3-0324 with only 1K SFT samples. We demonstrate that re-distillation can be used to efficiently balance multiple goals in RL. Our work explains several interesting phenomena in R1-style RL, shedding light on the mechanisms behind its empirical success. Code is available at: https://github.com/on1262/deep-reasoning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
