arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2303.16749cs.SEcs.AIcs.CLcs.LG
收藏

通过自然语言反馈训练改进代码生成

Improving Code Generation by Training with Natural Language Feedback

New York University(纽约大学);FAR AI;University of Sussex(萨塞克斯大学);University of the Basque Country UPV/EHU(巴斯克大学 UPV/EHU);Genentech(基因泰克);CIFAR LMB(加拿大高级研究院 LMB);Anthropic

Angelica Chen, Jérémy Scheurer, Tomasz Korbak, Jon Ander Campos, Jun Shern Chan, Samuel R. Bowman, Kyunghyun Cho, Ethan Perez

AI总结:

本文提出语言反馈模仿学习(ILF),在训练时利用少量人工自然语言反馈改进代码生成,使 Codegen-Mono 6.1B 在 MBPP 上 pass@1 相对提升 38%,优于仅基于示范的微调方法。

摘要:

预训练大语言模型(LLMs)在推理时利用自然语言反馈的潜力是近期令人兴奋的进展。我们基于这一观察,形式化了一种在训练时从自然语言反馈中学习的算法,称之为语言反馈模仿学习(ILF)。ILF 在训练期间仅需少量人工编写的反馈,并且在测试时不需要相同的反馈,因此既用户友好又样本高效。我们进一步表明,ILF 可被视为最小化与真实分布之间 KL 散度的一种形式,并在神经程序合成任务上展示了概念验证。我们使用 ILF 将 Codegen-Mono 6.1B 模型在 Mostly Basic Python Problems(MBPP)基准上的 pass@1 率相对提高了 38%(绝对提高 10%),优于在 MBPP 上微调以及在人工编写的修复程序上微调。总体而言,我们的结果表明,对于提升 LLM 在代码生成任务上的性能,从人工编写的自然语言反馈中学习比仅基于示范进行训练更有效且样本效率更高。

英文摘要:

The potential for pre-trained large language models (LLMs) to use natural language feedback at inference time has been an exciting recent development. We build upon this observation by formalizing an algorithm for learning from natural language feedback at training time instead, which we call Imitation learning from Language Feedback (ILF). ILF requires only a small amount of human-written feedback during training and does not require the same feedback at test time, making it both user-friendly and sample-efficient. We further show that ILF can be seen as a form of minimizing the KL divergence to the ground truth distribution and demonstrate a proof-of-concept on a neural program synthesis task. We use ILF to improve a Codegen-Mono 6.1B model's pass@1 rate by 38% relative (and 10% absolute) on the Mostly Basic Python Problems (MBPP) benchmark, outperforming both fine-tuning on MBPP and fine-tuning on repaired programs written by humans. Overall, our results suggest that learning from human-written natural language feedback is both more effective and sample-efficient than training exclusively on demonstrations for improving an LLM's performance on code generation tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑