arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2312.12450cs.SEcs.AIcs.LGcs.PL
收藏

它能编辑吗?评估大型语言模型遵循代码编辑指令的能力

Can It Edit? Evaluating the Ability of Large Language Models to Follow Code Editing Instructions

Northeastern University(东北大学);Wellesley College(韦尔斯利学院);HuggingFace;Roblox

Federico Cassano, Luisa Li, Akul Sethi, Noah Shinn, Abby Brennan-Jones, Jacob Ginesin, Edward Berman, George Chakhnashvili, Anton Lozhkov, Carolyn Jane Anderson, Arjun Guha

AI总结:

本文针对大型语言模型在指令性代码编辑任务上的研究不足,提出了一个代码编辑基准和一个宽松许可的训练数据集,评估了前沿模型并发现开放与封闭模型间存在差距,同时证明通过微调可显著提升开放模型的代码编辑能力。

摘要:

大量研究致力于开发和评估用于各种代码合成任务的大型语言模型。这些任务包括从自然语言合成代码、从代码合成测试以及合成代码的解释。相比之下,使用LLM进行指令性代码编辑的行为研究不足。这些任务是指向模型提供一段代码和一条修改代码的指令。编辑指令可能要求添加或删除功能、描述错误并要求修复,或者要求提供不同类型的解决方案。我们引入了一个精心制作的代码编辑任务基准,并用它评估了几个前沿的LLM。我们的评估揭示了最先进的开放模型和封闭模型之间存在显著差距。例如,即使是GPT-3.5-Turbo在代码编辑任务上也优于最佳开放模型。我们还引入了一个新的、经过精心筛选的、宽松许可的代码编辑任务训练数据集,并附带自然语言指令。使用这个训练数据集,我们展示了可以对开放Code LLMs进行微调,以显著提升其代码编辑能力,从而缩小开放模型与封闭模型之间的差距。所有代码、数据和模型均可在https://github.com/nuprl/CanItEdit获取。

英文摘要:

A significant amount of research is focused on developing and evaluating large language models for a variety of code synthesis tasks. These include synthesizing code from natural language, synthesizing tests from code, and synthesizing explanations of code. In contrast, the behavior of instructional code editing with LLMs is understudied. These are tasks in which the model is provided a block of code and an instruction to modify the code. The editing instruction may ask for a feature to be added or removed, describe a bug and ask for a fix, or ask for a different kind of solution. We introduce a carefully crafted benchmark of code editing tasks and use it to evaluate several cutting edge LLMs. Our evaluation exposes a significant gap between the capabilities of state-of-the-art open and closed models. For example, even GPT-3.5-Turbo is better than the best open model at code editing tasks. We also introduce a new, carefully curated, permissively licensed training dataset of code editing tasks coupled with natural language instructions. Using this training dataset, we show that we can fine-tune open Code LLMs to significantly improve their code editing capabilities, closing the gap between open and closed models. All code, data, and models are available at https://github.com/nuprl/CanItEdit.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑