ACECODER：通过自动化测试用例合成实现编码器强化学习
ACECODER: Acing Coder RL via Automated Test-Case Synthesis
AI总结：
本文提出ACECODER，通过自动化测试用例合成生成偏好对训练奖励模型，并用于强化学习，显著提升编码器模型性能，使7B模型媲美236B模型。
摘要：
近期编码器模型的大部分进展主要由监督微调（SFT）驱动，而强化学习（RL）的潜力在很大程度上仍未得到探索，这主要是由于代码领域缺乏可靠的奖励数据或模型。在本文中，我们通过利用自动化的大规模测试用例合成来增强代码模型训练，以应对这一挑战。具体而言，我们设计了一个流水线，从现有代码数据中生成大量的（问题，测试用例）对。利用这些测试用例，我们基于采样程序的通过率构建偏好对，以使用Bradley-Terry损失训练奖励模型。通过最佳32次采样，该方法在Llama-3.1-8B-Ins上平均提升10个百分点，在Qwen2.5-Coder-7B-Ins上提升5个百分点，使得7B模型与236B的DeepSeek-V2.5相当。此外，我们使用奖励模型和测试用例通过奖励进行强化学习，在HumanEval、MBPP、BigCodeBench和LiveCodeBench（V4）上均取得了一致的改进。值得注意的是，我们遵循R1风格的训练，直接从Qwen2.5-Coder-base开始，并表明我们的RL训练仅需80个优化步骤即可在HumanEval-plus上提升超过25%，在MBPP-plus上提升6%。我们相信，我们的结果凸显了强化学习在编码器模型中的巨大潜力。
英文摘要：
Most progress in recent coder models has been driven by supervised fine-tuning (SFT), while the potential of reinforcement learning (RL) remains largely unexplored, primarily due to the lack of reliable reward data/model in the code domain. In this paper, we address this challenge by leveraging automated large-scale test-case synthesis to enhance code model training. Specifically, we design a pipeline that generates extensive (question, test-cases) pairs from existing code data. Using these test cases, we construct preference pairs based on pass rates over sampled programs to train reward models with Bradley-Terry loss. It shows an average of 10-point improvement for Llama-3.1-8B-Ins and 5-point improvement for Qwen2.5-Coder-7B-Ins through best-of-32 sampling, making the 7B model on par with 236B DeepSeek-V2.5. Furthermore, we conduct reinforcement learning with both reward models and test-case pass rewards, leading to consistent improvements across HumanEval, MBPP, BigCodeBench, and LiveCodeBench (V4). Notably, we follow the R1-style training to start from Qwen2.5-Coder-base directly and show that our RL training can improve model on HumanEval-plus by over 25\% and MBPP-plus by 6\% for merely 80 optimization steps. We believe our results highlight the huge potential of reinforcement learning in coder models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
