arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2502.01715cs.SEcs.AI
收藏

面向代码生成的过程监督强化学习

Process-Supervised Reinforcement Learning for Code Generation

Beijing Institute of Technology(北京理工大学);Beijing Normal University(北京师范大学)

Yufan Ye, Ting Zhang, Wenbin Jiang, Hua Huang

AI总结:

本文提出基于语句变异/重构与编译执行验证的逐行过程监督数据构建方法,并集成到 PRLCoder 框架中,证明过程监督强化学习在代码生成任务上显著优于仅依赖结果监督的方法。

摘要:

现有基于结果监督的强化学习策略已被证明能够有效提升大语言模型(LLMs)在代码生成任务上的性能。尽管基于过程监督的强化学习在处理多步推理任务中展现出巨大潜力,但其在代码生成中的有效性仍未得到充分探索和论证。主要障碍在于构建高质量过程监督数据需要大量人工专业知识和计算资源,成本高昂。为应对这一挑战,我们提出了一种“语句变异/重构—编译与执行验证”策略:通过教师模型逐行变异和重构代码,并利用编译器执行结果自动标注每一行,从而生成逐行的过程监督数据,这对训练过程监督奖励模型至关重要。随后,将训练好的奖励模型集成到 PRLCoder 框架中,并在多个基准上进行了实验验证。实验结果表明,过程监督强化学习显著优于仅依赖结果监督的方法。值得注意的是,在处理复杂代码生成任务时,过程监督强化学习展现出明显优势,既能保证代码生成过程的完整性,又能确保生成结果的正确性。

英文摘要:

Existing reinforcement learning strategies based on outcome supervision have proven effective in enhancing the performance of large language models(LLMs) for code generation. While reinforcement learning based on process supervision has shown great promise in handling multi-step reasoning tasks, its effectiveness in code generation remains largely underexplored and underjustified. The primary obstacle stems from the resource-intensive nature of constructing high-quality process-supervised data, which demands substantial human expertise and computational resources. In response to this challenge, we propose a "statement mutation/refactoring-compile and execution verification" strategy: mutating and refactoring code line-by-line through a teacher model, and utilizing compiler execution results to automatically label each line, resulting in line-by-line process-supervised data, which is pivotal for training a process-supervised reward model. The trained reward model is then integrated into the PRLCoder framework, followed by experimental validation on several benchmarks. Experimental results demonstrate that process-supervised reinforcement learning significantly surpasses methods relying solely on outcome supervision. Notably, in tackling complex code generation tasks, process-supervised reinforcement learning shows a clear advantage, ensuring both the integrity of the code generation process and the correctness of the generation results.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑