arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2411.02462cs.SEcs.AIcs.LG
收藏

大型语言模型参数高效微调用于单元测试生成:一项实证研究

Parameter-Efficient Fine-Tuning of Large Language Models for Unit Test Generation: An Empirical Study

Norwegian University of Science and Technology(挪威科学技术大学)

André Storhaug, Jingyue Li

AI总结:

本文通过实证研究探讨了大型语言模型在单元测试生成中的参数高效微调方法,发现LoRA在某些情况下性能可与全量微调媲美,提示微调在成本效益上表现最佳。

摘要:

参数高效微调(PEFT)方法通过仅微调模型参数的子集,提供了一种有前景的解决方案,从而在降低大型语言模型(LLMs)微调计算成本的同时保持其性能。现有研究已探索了使用PEFT和LLMs进行各种代码相关任务,并发现PEFT技术的有效性依赖于任务。当前最先进的方法局限于使用全量微调的LLMs生成单元测试。PEFT技术在单元测试生成中的应用仍处于探索阶段。本文在十三种不同架构和规模的模型上调查了全量微调和多种PEFT方法,包括LoRA、(IA)^3和提示微调。我们使用已建立的基准数据集来评估它们在单元测试生成中的有效性,并测量生成测试的语法正确性、CodeBLEU、pass@1、指令覆盖、分支覆盖和突变分数。我们的发现表明,在某些情况下,LoRA在单元测试生成中的性能可以与全量微调相媲美。如果考虑训练成本,提示微调是最具成本效益的方法,特别是对于大型模型。然而,全量微调或PEFT微调的模型可能生成的可执行测试用例比基线模型更少,因为它们生成的测试调用不存在的方法或存在类型不匹配。对于生成的可执行测试用例,来自微调模型的测试覆盖优于基线模型。

英文摘要:

Parameter-efficient fine-tuning (PEFT) methods, which fine-tune only a subset of model parameters, offer a promising solution by reducing the computational costs of tuning large language models (LLMs) while maintaining their performance. Existing studies have explored using PEFT and LLMs for various code-related tasks and found that the effectiveness of PEFT techniques is task-dependent. The state-of-the-art is limited to using LLMs with full fine-tuning to generate unit tests. The application of PEFT techniques in unit test generation remains underexplored. This paper investigates both full fine-tuning and various PEFT methods, including LoRA, (IA)^3, and prompt tuning, across thirteen models of different architectures and sizes. We use well-established benchmark datasets to evaluate their effectiveness in unit test generation and measure syntax correctness, CodeBLEU, pass@1, instruction coverage, branch coverage, and mutation score of the generated tests. Our findings show that LoRA can deliver performance comparable to full fine-tuning for unit test generation in several cases. If training costs are valued, prompt tuning is the most cost-effective approach, particularly for large models. However, the models tuned with full fine-tuning or PEFT may generate fewer executable test cases than the baseline model because they generate more tests calling nonexistent methods or having type mismatches. For the generated ones that are executable, the ones from the tuned models show better test coverage than those from the baseline model.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑