arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2301.13816cs.LGcs.AIcs.CLcs.PL
收藏

基于执行的深度强化学习代码生成

Execution-based Code Generation using Deep Reinforcement Learning

Virginia Tech(弗吉尼亚理工学院)

Parshin Shojaee, Aneesh Jain, Sindhu Tipirneni, Chandan K. Reddy

AI总结:

针对现有代码生成方法忽略代码序列级特征的问题,提出结合预训练PL模型与PPO强化学习的PPOCoder框架,利用执行反馈优化模型,在多任务多语言上显著提升编译成功率和功能正确性。

摘要:

在大规模代码语料上预训练的编程语言(PL)模型被用于自动化软件工程流程,在简化代码补全、代码翻译、程序综合等各类代码生成任务方面展现出巨大潜力。然而,现有方法主要依赖从文本生成领域借鉴的监督微调目标,忽略了代码独特的序列级特征,包括但不限于可编译性以及语法和功能正确性。为解决这一局限,我们提出PPOCoder,这是一种新型代码生成框架,将预训练PL模型与近端策略优化(PPO,一种广泛使用的深度强化学习技术)协同结合。通过利用代码执行和结构对齐带来的不可微反馈,PPOCoder将外部代码专属知识无缝整合到模型优化过程中。值得注意的是,PPOCoder是任务无关且模型无关的框架,可用于不同代码生成任务和编程语言。我们在三项代码生成任务上开展了大量实验,结果表明,与SOTA方法相比,所提方法具备有效性,在不同编程语言的编译成功率和功能正确性上均实现显著提升。

英文摘要:

The utilization of programming language (PL) models, pre-trained on large-scale code corpora, as a means of automating software engineering processes has demonstrated considerable potential in streamlining various code generation tasks such as code completion, code translation, and program synthesis. However, current approaches mainly rely on supervised fine-tuning objectives borrowed from text generation, neglecting unique sequence-level characteristics of code, including but not limited to compilability as well as syntactic and functional correctness. To address this limitation, we propose PPOCoder, a new framework for code generation that synergistically combines pre-trained PL models with Proximal Policy Optimization (PPO) which is a widely used deep reinforcement learning technique. By utilizing non-differentiable feedback from code execution and structure alignment, PPOCoder seamlessly integrates external code-specific knowledge into the model optimization process. It's important to note that PPOCoder is a task-agnostic and model-agnostic framework that can be used across different code generation tasks and PLs. Extensive experiments on three code generation tasks demonstrate the effectiveness of our proposed approach compared to SOTA methods, achieving significant improvements in compilation success rates and functional correctness across different PLs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑