arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.08873cs.LGstat.APstat.ML
收藏

CoDistill-GRPO:一种高效的群体相对策略优化共蒸馏方法

CoDistill-GRPO: A Co-Distillation Recipe for Efficient Group Relative Policy Optimization

University of Michigan, Ann Arbor(密歇根大学,安阿伯分校);Google Research(谷歌研究院);Google DeepMind(谷歌DeepMind)

Soo Min Kwon, Ziteng Sun, Ananda Theertha Suresh, Himanshu Jain, Sanjiv Kumar

AI总结:

本文提出CoDistill-GRPO,通过同时训练大模型和小模型,利用精心设计的GRPO目标提升小模型性能,减少计算开销,在数学基准上显著优于标准GRPO。

摘要:

群体相对策略优化(GRPO)已成为提升语言模型推理能力的强大算法,但常因困难任务的稀疏奖励而无法改进小型模型。现有方法通过利用更大模型提供提示或通过知识蒸馏(KD)提供密集奖励信号来缓解这一问题。然而,这假设存在这样的Oracle,而训练一个此类模型会显著增加总训练时间。在本文中,我们提出CoDistill-GRPO,一种共蒸馏算法,通过最大化精心设计的GRPO目标同时训练大模型和小模型。两个模型相互学习:小模型使用基于策略的KD奖励从大模型的分布中学习,而大模型通过小模型生成的rollouts进行更新,并通过重要性重加权减少rollout生成的计算开销。我们展示了CoDistill-GRPO在数学基准上显著优于标准GRPO,针对Qwen和Llama模型。具体而言,使用Qwen2.5-Math-1.5B模型,我们在Minerva数据集上观察到基模型的准确率提高了超过11.6个百分点,并在GRPO上额外提高了6.0个百分点。有趣的是,通过CoDistill-GRPO训练的较大模型(Qwen2.5-Math-7B)在使用小模型rollouts时几乎与标准GRPO性能相当。这表明CoDistill-GRPO是标准GRPO的更经济的选择,为较大模型提供约18%的加速,这可能具有独立的兴趣。

英文摘要:

Group Relative Policy Optimization (GRPO) has emerged as a powerful algorithm for improving the reasoning capabilities of language models, but often fails to improve small models due to sparse rewards on difficult tasks. Existing works mitigate this issue by leveraging a larger model, either to provide hints for rollouts or to provide dense reward signals through knowledge distillation (KD). However, this assumes the existence of such an oracle, and training one can significantly increase total training time. In this work, we propose CoDistill-GRPO, a co-distillation algorithm that simultaneously trains a large and a small model by maximizing carefully designed GRPO objectives. The two models learn from each other: the small model uses an on-policy KD reward to learn from the large model's distribution, while the large model is updated using rollouts generated by the small model with importance reweighting, reducing the computational overhead of rollout generation. We show that CoDistill-GRPO substantially improves small model performance over standard GRPO on mathematical benchmarks across both Qwen and Llama models. Specifically, with Qwen2.5-Math-1.5B, we observe an accuracy increase of over 11.6 percentage points over the base model and an additional 6.0 percentage points over GRPO on the Minerva dataset. Interestingly, the larger model (Qwen2.5-Math-7B) trained with CoDistill-GRPO nearly matches standard GRPO performance despite training on small-model rollouts. This highlights CoDistill-GRPO as a cost-effective alternative to GRPO for larger models, yielding an approximate 18% speedup, which may be of independent interest.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑