arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2306.09896cs.CLcs.AIcs.PLcs.SE
收藏

自修复是代码生成的银弹吗?

Is Self-Repair a Silver Bullet for Code Generation?

MIT CSAIL(麻省理工学院计算机科学与人工智能实验室);Microsoft Research(微软研究院)

Theo X. Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, Armando Solar-Lezama

AI总结:

该研究分析了Code Llama、GPT-3.5、GPT-4在HumanEval和APPS数据集上的代码自修复能力,发现考虑成本时自修复增益有限,其瓶颈在于模型自我反馈能力,且远不及人类水平调试的效果。

摘要:

大语言模型在代码生成方面已展现出卓越的能力,但在执行复杂任务时仍存在困难。自修复(即模型调试并修复自身代码的技术)近期已成为提升这类场景下性能的流行方法。然而,尽管自修复的应用日益广泛,现有相关研究的范围仍较为有限;因此在许多场景中,其功效仍未得到充分明晰。本文中,我们分析了Code Llama、GPT-3.5和GPT-4在HumanEval和APPS数据集中的问题上执行自修复的能力。我们发现,当将执行修复的成本纳入考量时,性能提升通常较为有限,在数据子集间的差异很大,有时甚至完全不存在提升。我们假设这是因为自修复受限于模型对自身代码提供反馈的能力;通过使用更强的模型人为提升反馈质量,我们观察到了大幅提升的性能增益。同样,一项让GPT-4获取人类参与者反馈的小规模研究表明,即便对于最强的模型,自修复的效果仍远落后于人类水平调试所能达到的效果。

英文摘要:

Large language models have shown remarkable aptitude in code generation, but still struggle to perform complex tasks. Self-repair -- in which the model debugs and repairs its own code -- has recently become a popular way to boost performance in these settings. However, despite its increasing popularity, existing studies of self-repair have been limited in scope; in many settings, its efficacy thus remains poorly understood. In this paper, we analyze Code Llama, GPT-3.5 and GPT-4's ability to perform self-repair on problems taken from HumanEval and APPS. We find that when the cost of carrying out repair is taken into account, performance gains are often modest, vary a lot between subsets of the data, and are sometimes not present at all. We hypothesize that this is because self-repair is bottlenecked by the model's ability to provide feedback on its own code; using a stronger model to artificially boost the quality of the feedback, we observe substantially larger performance gains. Similarly, a small-scale study in which we provide GPT-4 with feedback from human participants suggests that even for the strongest models, self-repair still lags far behind what can be achieved with human-level debugging.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑