$\mathcal{B}$-Coder：基于价值的深度强化学习用于程序合成
$\mathcal{B}$-Coder: Value-Based Deep Reinforcement Learning for Program Synthesis
AI总结：
本文提出$\mathcal{B}$-Coder，一种基于价值的深度强化学习方法，用于程序合成，通过初始化协议和保守Bellman算子降低训练复杂度，并利用价值函数后处理生成程序，实现了最先进的性能。
摘要：
程序合成旨在根据问题规格说明，特别是我们语境中的自然语言描述，创建准确、可执行的程序。近期研究利用强化学习（RL）与大型语言模型（LLMs）相结合的力量，显著提升了代码生成能力。强化学习的应用侧重于直接优化功能正确性，相比传统监督方法具有优势。尽管基于策略的强化学习方法在程序合成的强化学习文献中占主导地位，但程序合成任务的性质暗示其与基于价值的方法自然契合。这源于丰富的离策略程序集合，包括人类程序员开发的程序以及历史样本，加之通过自动化单元测试对生成程序进行直接验证，意味着奖励易于获取。与主流使用的基于策略算法不同，我们的工作探索了基于价值方法的可行性，从而开发出我们的$\mathcal{B}$-Coder（读作Bellman coder）。然而，由于程序合成固有的巨大搜索空间，训练基于价值的方法面临挑战。为此，我们引入了利用预训练语言模型的强化学习智能体初始化协议和保守的Bellman算子，以降低训练复杂度。此外，我们展示了如何利用学习到的价值函数作为双重策略来后处理生成的程序。我们的实证评估表明，$\mathcal{B}$-Coder在与基于策略的方法相比时达到了最先进的性能。值得注意的是，这一成就是在极小的奖励工程努力下实现的，凸显了基于价值的强化学习的有效性，且独立于奖励设计。
英文摘要：
Program synthesis aims to create accurate, executable programs from problem specifications, specifically from natural language descriptions in our context. Recent studies have leveraged the power of reinforcement learning (RL) in conjunction with large language models (LLMs), significantly enhancing code generation capabilities. The application of RL focuses on directly optimizing for functional correctness, offering an advantage over conventional supervised methods. Despite policy-based RL methods dominating the literature on RL for program synthesis, the nature of program synthesis tasks hints at a natural alignment with value-based methods. This stems from the rich collection of off-policy programs, including those developed by human programmers and also historical samples, coupled with the straightforward verification of generated programs through automated unit testing, meaning rewards are easy to obtain. Diverging from the dominant use of policy-based algorithms, our work explores the feasibility of value-based approaches, leading to the development of our $\mathcal{B}$-Coder (pronounced Bellman coder). Yet, training value-based methods presents challenges due to the enormous search space inherent to program synthesis. To this end, we introduce an initialization protocol for RL agents utilizing pre-trained LMs and a conservative Bellman operator to reduce training complexities. Moreover, we demonstrate how to leverage the learned value functions as a dual strategy to post-process generated programs. Our empirical evaluations demonstrated $\mathcal{B}$-Coder's capability in achieving state-of-the-art performance when compared to policy-based methods. Remarkably, this achievement is reached with minimal reward engineering effort, highlighting the effectiveness of value-based RL, independent of reward designs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
