arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.21180cs.LGcs.SE
收藏

用于密集奖励的领域可适应强化学习代码生成

Domain-Adaptable Reinforcement Learning for Code Generation with Dense Rewards

Hessian Center for Artificial Intelligence (hessian.AI)(海斯曼人工智能中心);National Research Center for Applied Cybersecurity ATHENE(应用网络安全国家研究中心ATHENE)

Erfan Aghadavoodi Jolfaei, Daniel Maninger, Abhinav Anand, Mert Tiftikci, Mira Mezini

AI总结:

本研究提出了一种领域可适应的强化学习框架,用于改进代码生成的正确性、质量和安全性,通过定制化的执行感知奖励公式和令牌级奖励映射机制,提高了代码生成在不同领域中的适应性和执行效率。

摘要:

大型语言模型在自动化代码生成中显示出强大的潜力,但缺乏正确性、质量和安全性的保证,特别是在领域特定约束方面。例如在机器人领域,代码生成越来越多地用于规划和执行动作,环境意识和物理约束至关重要。为了促进代码生成LLM适应多样化需求,包括领域特定需求,我们提出了一种强化学习框架,通过近端策略优化微调预训练LLM。我们的可定制执行感知奖励公式捕捉并优化语法、功能正确性、代码风格、安全性和模拟器可执行性。一个令牌级奖励映射机制使从执行结果到生成令牌的有效信用分配成为可能。该框架在通用代码生成(MBPP/MBPP+)和机器人程序合成(RoboEval)上进行了评估。结果表明,在功能正确性和模拟器可执行性方面有显著改进,包括在MBPP上的pass@1绝对增加19%,在RoboEval上的执行失败减少51%。这些发现表明,结构化的强化学习可以有效地将语言模型对齐到正确的程序生成和领域特定需求。

英文摘要:

Large language models show strong potential for automated code generation, but lack guarantees for correctness, quality, safety, and domain-specific constraints. For instance in robotics, where code generation is increasingly being used for planning and executing actions, awareness of the environment and physical constraints is critical. To facilitate the adaption of code-generating LLMs to diverse requirements, including domain-specific ones, we present a reinforcement learning framework that fine-tunes pre-trained LLMs using proximal policy optimization. Our customizable execution-aware reward formula captures and optimizes syntax, functional correctness, code style, security, and simulator executability. A token-level reward mapping mechanism enables effective credit assignment from execution outcomes to generated tokens. The framework is evaluated on general-purpose code generation (MBPP/MBPP+) and robotic program synthesis (RoboEval). The results show substantial improvements in functional correctness and simulator executability, including an absolute pass@1 increase of 19% on MBPP and a reduction in execution failures by 51% on RoboEval. These findings demonstrate that structured reinforcement learning can effectively align language models to correct program generation and domain-specific requirements.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑