用于训练准确代码生成器的LLM辅助代码清理
LLM-Assisted Code Cleaning For Training Accurate Code Generators
AI总结:
本研究提出一种基于LLM的代码清理流水线,通过重命名变量、模块化分解及插入自然语言计划提升代码质量,在算法基准上微调CodeLLaMa-7B使性能提升达30%,并证明少量高质量数据优于全量原始数据。
摘要:
自然语言到代码的生成是LLMs的一个重要应用领域,并受到了社区的广泛关注。大多数相关研究仅集中于增加训练集的数量和功能正确性,而忽略了程序的其他风格元素。最近,数据质量引起了极大的兴趣,多项研究展示了其对提升性能的重要性。在本研究中,我们探讨了代码的数据质量,发现使代码更具结构化和可读性能够提升系统的代码生成性能。我们构建了一个新颖的数据清理流水线,该流水线利用这些原则通过以下方式转换现有程序:1.) 重命名变量,2.) 模块化并将复杂代码分解为更小的辅助子函数,以及 3.) 通过基于LLM的转换插入基于自然语言的计划。我们在两个具有挑战性的算法代码生成基准上评估了我们的方法,并发现在我们转换后的模块化程序上微调CodeLLaMa-7B,与在原始数据集上微调相比,性能提升了高达30%。此外,我们证明了使用更少量但更高质量的数据可以带来性能提升,发现在整个原始数据集上微调的模型,其表现不如仅在我们清理后数据集的15%上训练的模型。即使与闭源模型相比,我们的模型也优于大得多的AlphaCoder模型。
英文摘要:
Natural language to code generation is an important application area of LLMs and has received wide attention from the community. The majority of relevant studies have exclusively concentrated on increasing the quantity and functional correctness of training sets while disregarding other stylistic elements of programs. More recently, data quality has garnered a lot of interest and multiple works have showcased its importance for improving performance. In this work, we investigate data quality for code and find that making the code more structured and readable leads to improved code generation performance of the system. We build a novel data-cleaning pipeline that uses these principles to transform existing programs by 1.) renaming variables, 2.) modularizing and decomposing complex code into smaller helper sub-functions, and 3.) inserting natural-language based plans via LLM based transformations. We evaluate our approach on two challenging algorithmic code generation benchmarks and find that fine-tuning CodeLLaMa-7B on our transformed modularized programs improves the performance by up to 30% compared to fine-tuning on the original dataset. Additionally, we demonstrate improved performance from using a smaller amount of higher-quality data, finding that a model fine-tuned on the entire original dataset is outperformed by a model trained on 15% of our cleaned dataset. Even in comparison to closed-source models, our models outperform the much larger AlphaCoder models.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。