用于代码合成的自动单元测试数据生成与Actor-Critic强化学习
Automatic Unit Test Data Generation and Actor-Critic Reinforcement Learning for Code Synthesis
AI总结：
针对代码合成中单元测试奖励信号难以获取的问题，提出自动生成函数签名与单元测试数据的方法，并设计Actor-Critic强化学习训练方案，使预训练代码模型性能显著提升。
摘要：
大型预训练语言模型在代码合成领域的出现，在各项基准上展现了卓越性能，其将代码生成问题以类似于自然语言生成的方式处理，并使用语言建模（LM）目标进行训练。此外，编程语言代码在语义上具有精确可评估性的属性——通过使用单元测试来检查其功能正确性——这使得强化学习（RL）可作为一种进一步的训练范式。先前的研究表明，RL可以这样被应用以提升模型的编程能力；然而，这类基于RL的方法依赖于由预定义的单元测试所提供的奖励信号，与LM目标中使用的海量爬取代码数据集相比，这些单元测试的获取难度要大得多。在本研究中，我们提出了一种新颖的方法，能够自动获取由函数签名及相关联的单元测试组成的数据，适用于代码合成模型的RL训练。我们还引入了一种简单直接但有效的Actor-Critic RL训练方案，并表明该方案与自动生成的训练数据相结合，能使预训练代码语言模型的性能获得提升，比原始的底层代码合成LM最高提升9.9%，比使用标准PPO或CodeRL训练的基于RL的模型最高提升4.3%。
英文摘要：
The advent of large pre-trained language models in the domain of Code Synthesis has shown remarkable performance on various benchmarks, treating the problem of Code Generation in a fashion similar to Natural Language Generation, trained with a Language Modelling (LM) objective. In addition, the property of programming language code being precisely evaluable with respect to its semantics -- through the use of Unit Tests to check its functional correctness -- lends itself to using Reinforcement Learning (RL) as a further training paradigm. Previous work has shown that RL can be applied as such to improve models' coding capabilities; however, such RL-based methods rely on a reward signal based on defined Unit Tests, which are much harder to obtain compared to the huge crawled code datasets used in LM objectives. In this work, we present a novel approach to automatically obtain data consisting of function signatures and associated Unit Tests, suitable for RL training of Code Synthesis models. We also introduce a straightforward, simple yet effective Actor-Critic RL training scheme and show that it, in conjunction with automatically generated training data, leads to improvement of a pre-trained code language model's performance by up to 9.9% improvement over the original underlying code synthesis LM, and up to 4.3% over RL-based models trained with standard PPO or CodeRL.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
