arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2511.06307cs.LG
收藏

DRIVE:面向竞争性代码生成中可验证奖励的强化学习的数据策展最佳实践

DRIVE: Data Curation Best Practices for Reinforcement Learning with Verifiable Reward in Competitive Code Generation

Hunyuan Team, Tencent(腾讯 Hunyuan 团队)

Speed Zhu, Jianwei Cai, Guang Chen, Lulu Wu, Saiyong Yang, Wiggin Zhou

AI总结:

本文提出DRIVE框架,通过两阶段RLVR训练(GRPO加Pre-GRPO)和课程设计优化竞争性编程代码生成,在Qwen2.5-32B上取得同等规模最优性能。

摘要:

近期以推理为先的模型(如 OpenAI o1、DeepSeek R1)重新激发了对带可验证奖励的强化学习(RLVR)的研究兴趣。然而,当前进展主要由数学任务(如 AIME)主导,竞争性编程代码生成领域探索不足,且数据策展受到的关注少于强化学习算法设计。我们研究了如何构建 RLVR 数据集(即 RL 提示),并提出了能在竞争性编程代码生成上取得优异性能的实用训练技术。我们的流程首先利用从强大开源模型蒸馏得到的监督微调(SFT),并补充通用型和推理密集型数据。随后,强化学习采用两阶段过程,使用可执行、由测试用例驱动的奖励:第一阶段,在一个大规模、均匀分布的竞争性编程问题集上训练,使用分组相对策略优化(GRPO),每个提示采样 8 个 rollout,并采用相对较短的响应生成长度窗口(例如 SFT 阶段为 32k,本阶段为 24k),以扩大熵并缓解重复和截断问题;第二阶段,我们执行 Pre-GRPO:在一个小规模、高质量的难题集上更新,使用大规模 rollout 预算(每个提示 64 个 rollout),并采用硬聚焦课程策略,在训练过程中持续保留最困难的实例。我们在 Qwen2.5-32B 上实现了该方法,并在 LeetCode 和 Codeforces 周赛上进行评估以避免数据泄漏。最终得到的模型在同等规模模型中取得了最先进的性能,可与 DeepSeek v3.1 和 Doubao-1.5-Thinking 等领先系统相媲美。我们还考察了扩展趋势,并在内部大规模 MoE 模型上观察到显著的强化学习扩展效应。我们的研究提炼出了面向竞争性编程代码生成的 RLVR 中关于数据策展、熵扩展和课程设计的简明最佳实践。

英文摘要:

Recent reasoning-first models (e.g., OpenAI o1, DeepSeek R1) have spurred a resurgence of interest in RLVR. Nevertheless, advances are dominated by mathematics (e.g., AIME), with competitive-programming code generation underexplored and data curation receiving less attention than RL algorithm design. We investigate how to construct RLVR datasets (i.e., RL prompts) and present practical training techniques that yield strong performance on competitive-programming code generation. Our pipeline begins with supervised fine-tuning (SFT) distilled from strong open-source models, augmented with general-purpose and reasoning-intensive data. RL then follows a two-stage process with executable, testcase-driven rewards: first, training on a large, uniformly distributed set of competitive-programming problems using Group Relative Policy Optimization (GRPO) with 8 rollouts per prompt and a relatively short response-generation window (e.g., 32k during SFT and 24k in this stage) to expand entropy and mitigate repetition and truncation; second, we perform \textbf{Pre-GRPO}: updating on a small, high-quality set of challenging problems with a large rollout budget (64 rollouts per prompt) under a hard-focus curriculum that continuously retains the most difficult instances throughout training. We implement our method on Qwen2.5-32B and evaluate on LeetCode and Codeforces weekly contests to avoid data leakage. The resulting model achieves state-of-the-art performance among models of similar scale and is comparable to leading systems such as DeepSeek v3.1 and Doubao-1.5-Thinking. We also examine scaling trends and observe strong RL scaling on an internal large-scale MoE model. Our study distills concise best practices for data curation, entropy expansion, and curriculum design in RLVR for competitive-programming code generation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑