通过强化重排序实现代码生成的迭代自训练
Iterative Self-Training for Code Generation via Reinforced Re-Ranking
AI总结:
针对代码生成模型输出随机性强、易出错的问题,提出基于近端策略优化(PPO)的迭代自训练重排序方法,通过迭代优化训练数据集提升重排序准确性,13.4B参数模型在MultiPL-E数据集上质量优于33B模型且速度快三倍,性能与GPT-4相当并在一种语言上超越。
摘要:
生成能解决复杂编程任务的高质量代码具有挑战性,尤其是当前基于解码器的模型会产生高度随机的输出。在代码生成中,即使微小错误也容易破坏整个解决方案。利用多个采样的解决方案可显著提升整体输出质量。增强代码生成的一种有效方法是将代码生成模型与重排序模型配对,后者从生成的样本中选择最佳解决方案。我们提出一种新颖的迭代自训练方法,通过近端策略优化(PPO)自训练重排序模型,旨在同时提高重排序准确性和整体代码生成过程。与传统PPO方法(聚焦于用奖励模型优化生成模型)不同,我们的方法强调开发鲁棒的奖励/重排序模型。该模型通过重排序提升生成代码的质量,并解决奖励模型在PPO与重排序器对齐过程中可能忽略的问题和错误。我们的方法通过重新评估输出、识别高评分负例并将其纳入训练循环来迭代优化训练数据集,从而提升模型性能。我们在MultiPL-E数据集上的评估显示,我们的13.4B参数模型在代码生成质量上优于33B模型,同时速度快三倍;此外,其性能与GPT-4相当,并在一种编程语言上超越了GPT-4。
英文摘要:
Generating high-quality code that solves complex programming tasks is challenging, especially with current decoder-based models that produce highly stochastic outputs. In code generation, even minor errors can easily break the entire solution. Leveraging multiple sampled solutions can significantly improve the overall output quality. One effective way to enhance code generation is by pairing a code generation model with a reranker model, which selects the best solution from the generated samples. We propose a novel iterative self-training approach for self-training reranker models using Proximal Policy Optimization (PPO), aimed at improving both reranking accuracy and the overall code generation process. Unlike traditional PPO approaches, where the focus is on optimizing a generative model with a reward model, our approach emphasizes the development of a robust reward/reranking model. This model improves the quality of generated code through reranking and addresses problems and errors that the reward model might overlook during PPO alignment with the reranker. Our method iteratively refines the training dataset by re-evaluating outputs, identifying high-scoring negative examples, and incorporating them into the training loop, that boosting model performance. Our evaluation on the MultiPL-E dataset demonstrates that our 13.4B parameter model outperforms a 33B model in code generation quality while being three times faster. Moreover, it achieves performance comparable to GPT-4 and surpasses it in one programming language.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。