$\ extbf{PLUM}$:通过由合成测试用例驱动的执行引导同策略偏好学习来改进代码语言模型
$\textbf{PLUM}$: Improving Code LMs with Execution-Guided On-Policy Preference Learning Driven By Synthetic Test Cases
AI总结:
针对代码语言模型监督微调无法区分代码对错的问题,提出同策略偏好学习框架PLUM,通过自动生成测试用例并评估候选代码构建偏好数据,免除了奖励模型训练,在多个基准上显著提升了通过率。
摘要:
偏好学习为解决监督微调(SFT)在代码语言模型中的局限性提供了一种有前景的方案,因为在监督微调中,模型并未被显式训练以区分正确与错误的代码。近期的研究表明,同策略数据是成功进行偏好学习的关键,即偏好数据是使用正在训练的同一策略语言模型收集的。受此启发,我们提出了 PLUM,这是一个为代码语言模型(code L$\ extbf{M}$s)增强测试用例的同策略偏好学习框架。该框架分为三个关键阶段运行:(1)从自然语言指令中自动生成测试用例;(2)通过评估从策略中采样的候选代码解决方案来创建偏好数据,这些数据随后可用于(3)训练策略语言模型。PLUM 免去了训练奖励模型的需求,从而允许进行大规模的同策略和在线偏好数据整理。PLUM 在标准基准(HumanEval、MBPP)以及更具挑战性的基准上进行了评估,相较于原始经过 SFT 的模型及其他执行反馈驱动的方法,取得了显著的改进。我们证明,即使在各种广泛使用的代码语言模型已经通过 SFT 充分训练的情况下,PLUM 的优势依然保持一致。例如,PLUM 在标准基准上的平均通过率最高提升了 4.8%,在 LiveCodeBench 上提升了 11.8%,展示了其有效性与泛化能力。我们还通过全面的实验证明了同策略与在线偏好学习的优势。
英文摘要:
Preference learning provides a promising solution to address the limitations of supervised fine-tuning (SFT) for code language models, where the model is not explicitly trained to differentiate between correct and incorrect code. Recent findings demonstrate that on-policy data is the key to successful preference learning, where the preference data is collected using the same policy LM being trained. Inspired by this, we propose PLUM, an on-policy $\textbf{P}$reference $\textbf{L}$earning framework A$\textbf{u}$gmented with test cases for code L$\textbf{M}$ s. The framework operates in three key stages: (1) automatic generation of test cases from natural language instructions, (2) creation of a preference data by evaluating candidate code solutions sampled from the policy, which can then be used to (3) train the policy LM. PLUM levitates the need to train reward models, allowing for large scale on-policy and online preference data collation. PLUM is evaluated on both standard benchmarks (HumanEval, MBPP) and more challenging ones (LiveCodeBench), delivering substantial improvements over original SFT'ed models and other execution-feedback-driven approaches. We show PLUM's benefits are consistent across various widely-used code LMs even they have been well-trained with SFT. For example, PLUM increases pass rates by up to 4.8% on average on standard benchmarks and 11.8% on LiveCodeBench, demonstrating its effectiveness and generalizability. We also demonstrate the benefits of on-policy and online preference learning by comprehensive experimentation.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。