基于故障代码驱动的测试用例合成与密集奖励塑造的鲁棒代码强化学习
Robust Code RL via Faulty-Code-Driven Test case Synthesis and Dense Reward Shaping
- Zhejiang University（浙江大学）
- Ant Group（蚂蚁集团）
机构由 AI 辅助整理，请以论文原文为准。
中文总结 AI 辅助
该研究提出RobustTests框架，通过故障代码驱动的测试用例合成与密集奖励塑造，实现RL微调的Qwen3-32B在LiveCodeBench上较基线方法获绝对3%性能提升，增强了LLM代码生成能力。
中文摘要 AI 辅助
可验证奖励强化学习（RLVR）已成为增强大型语言模型（LLM）代码生成能力的关键技术。然而，RLVR在代码实现中的有效性根本上受限于测试用例的全面性，因为代码验证中不足的测试覆盖率常导致误报，进而引发奖励黑客行为和策略退化。为缓解当前自动生成方法质量欠佳导致的奖励偏差，我们提出RobustTests框架，该框架引入故障代码驱动的测试用例合成策略，利用“接近正确的故障代码”引导模型精准捕捉潜在逻辑差异，并将验证器智能体与行为特征聚类相结合，以实现对无效和冗余测试用例的细粒度过滤。为解决合成测试用例中固有幻觉噪声导致的漏报问题，RobustTests还纳入基于通过率的分步密集奖励函数，通过细粒度反馈提升训练鲁棒性。通过该流程，我们构建了高质量数据集，扩充了CodeContests中的测试用例，涵盖更广泛的故障代码场景并显著提升诊断效用。实验结果表明，通过利用CodeContests中中等难度的问题子集进行训练，经RobustTests进行RL微调的Qwen3-32B在LiveCodeBench基准测试中相比基线方法实现了绝对3%的性能提升，证实了RobustTests框架在提升LLM代码生成能力方面的有效性。
英文摘要
Reinforcement Learning from Verifiable Rewards (RLVR) is pivotal for enhancing LLM code generation, yet its efficacy is often hindered by insufficient test case coverage, leading to reward hacking and policy degradation. To address this, we propose RobustTests, a framework featuring a faulty-code-driven test case synthesis strategy. By leveraging "near-correct" faulty codes, RobustTests captures latent logical discrepancies and employs validator agents with behavioral feature clustering to filter invalid or redundant test cases. Additionally, a stepwise dense reward function based on pass rates is introduced to mitigate false negatives and enhance training robustness. Using this pipeline, we construct an augmented version of the CodeContests+ dataset with superior diagnostic utility. Experimental results show that RL fine-tuning of Qwen3-32B via RobustTests achieves a 3% absolute gain on LiveCodeBench, demonstrating its effectiveness in advancing LLM code generation proficiency. Codes and data are available at https://huggingface.co/datasets/sid6/RobustTests.
