通过验证反馈的强化学习改进用于代码生成的小型语言模型
Improving Small Language Models for Code Generation with Reinforcement Learning from Verification Feedback
AI总结：
本研究通过基于验证奖励的强化学习（RLVR）结合LoRA微调，在MBPP基准上对小型语言模型（Qwen3-0.6B和Llama3.2-1B）进行Python代码生成实验，发现奖励设计对功能正确性和行为诊断有显著影响，并提出组合奖励可稳定权衡正确性与风格约束。
摘要：
具有可验证奖励的强化学习（RLVR）使用程序可检查的信号（如单元测试结果）训练语言模型，从而能够直接优化代码生成的功能正确性。我们在MBPP基准上使用两个小型模型（Qwen3-0.6B和Llama3.2-1B）结合LoRA微调，对Python代码生成进行了RLVR的实证研究。在多种奖励公式（如：仅单元测试奖励、通过Ruff linter的仅静态分析塑形、以及组合奖励）下，我们比较了基于组的策略优化变体（GRPO和GSPO），并评估了功能正确性和行为诊断。在我们的实验设置中，RLVR在提出的组合奖励配置下将MBPP测试上的pass@1提高了最多13个百分点。然而，我们发现奖励塑形可能引发系统性行为偏移：仅使用静态分析惩罚可能使策略偏向于生成更短的代码，从而减少lint错误，但未能可靠地提高功能正确性。相比之下，组合奖励减轻了这种退化，并在正确性和风格约束之间产生了更稳定的权衡。总体而言，我们的结果强调，RLVR在代码生成中的有效性对奖励设计和优化粒度高度敏感，并且除了pass@1之外的诊断指标（包括生成长度、Ruff严重性分布和执行错误类型）有助于识别失败模式。
英文摘要：
Reinforcement learning with verifiable rewards (RLVR) trains language models using programmatically checkable signals such as unit-test outcomes, enabling direct optimization for functional correctness in code generation. We conduct an empirical study of RLVR for Python code generation on the MBPP benchmark using two small models (Qwen3-0.6B and Llama3.2-1B) with LoRA fine-tuning. Across multiple reward formulations such as: unit-test-only rewards, static-analysis-only shaping via the Ruff linter, and a combined reward, we compare group-based policy optimization variants (GRPO and GSPO) and evaluate both functional correctness and behavioral diagnostics. In our experimental setting, RLVR improves pass@1 on MBPP test by up to 13 percentage points under proposed combined reward configuration. However, we find that reward shaping can induce systematic behavioral shifts: using only static-analysis penalties may bias the policy toward shorter completions that reduce lint errors without reliably improving functional correctness. In contrast, combined rewards mitigate this degeneration and yield more stable trade-offs between correctness and style constraints. Overall, our results highlight that RLVR effectiveness for code generation is highly sensitive to reward design and optimization granularity, and that diagnostics beyond pass@1, including generation length, Ruff severity profiles, and execution error types are useful for identifying failure modes.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
