arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.01391cs.SEcs.CL
收藏

StepCoder:利用编译器反馈的强化学习改进代码生成

StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedback

Fudan University(复旦大学);Huazhong University of Science and Technology(华中科技大学);KTH Royal Institute of Technology(皇家理工学院)
Shihan Dou, Yan Liu, Haoxiang Jia, Limao Xiong, Enyu Zhou, Wei Shen, Junjie Shan, Caishuang Huang, Xiao Wang, Xiaoran Fan, Zhiheng Xi, Yuhao Zhou, Tao Ji, Rui Z… 展开作者

Shihan Dou, Yan Liu, Haoxiang Jia, Limao Xiong, Enyu Zhou, Wei Shen, Junjie Shan, Caishuang Huang, Xiao Wang, Xiaoran Fan, Zhiheng Xi, Yuhao Zhou, Tao Ji, Rui Zheng, Qi Zhang, Xuanjing Huang, Tao Gui

AI总结:

StepCoder提出一种结合编译器反馈的强化学习框架,通过课程式代码补全子任务和细粒度优化解决长代码探索难题,并构建APPS+数据集,在基准测试中超越现有方法。

摘要:

大型语言模型(LLMs)的进步极大地推动了代码生成领域的发展。以往的工作将强化学习(RL)与编译器反馈相结合,用于探索LLMs的输出空间,以提升代码生成质量。然而,LLMs针对复杂人类需求生成的长代码使得RL探索面临挑战。此外,由于单元测试可能无法覆盖复杂代码,利用这些未执行的代码片段来优化LLMs是低效的。为应对这些挑战,我们引入了StepCoder,一种新颖的用于代码生成的RL框架,包含两个主要组件:CCCS通过将长序列代码生成任务分解为代码补全子任务的课程(Curriculum of Code Completion Subtasks)来解决探索挑战,而FGO则通过掩蔽未执行的代码段来提供细粒度优化(Fine-Grained Optimization),仅优化模型。此外,我们还构建了APPS+数据集用于RL训练,该数据集经过人工验证以确保单元测试的正确性。实验结果表明,我们的方法提升了探索输出空间的能力,并在相应基准测试中超越了最先进的方法。我们的数据集APPS+和StepCoder已在线提供。

英文摘要:

The advancement of large language models (LLMs) has significantly propelled the field of code generation. Previous work integrated reinforcement learning (RL) with compiler feedback for exploring the output space of LLMs to enhance code generation quality. However, the lengthy code generated by LLMs in response to complex human requirements makes RL exploration a challenge. Also, since the unit tests may not cover the complicated code, optimizing LLMs by using these unexecuted code snippets is ineffective. To tackle these challenges, we introduce StepCoder, a novel RL framework for code generation, consisting of two main components: CCCS addresses the exploration challenge by breaking the long sequences code generation task into a Curriculum of Code Completion Subtasks, while FGO only optimizes the model by masking the unexecuted code segments to provide Fine-Grained Optimization. In addition, we furthermore construct the APPS+ dataset for RL training, which is manually verified to ensure the correctness of unit tests. Experimental results show that our method improves the ability to explore the output space and outperforms state-of-the-art approaches in corresponding benchmarks. Our dataset APPS+ and StepCoder are available online.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑