arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.00905cs.SE
收藏

基于大语言模型的代码审查自动化的微调与提示工程

Fine-Tuning and Prompt Engineering for Large Language Models-based Code Review Automation

Chanathip Pornprasit, Chakkrit Tantithamthavorn

AI总结:

本文比较GPT-3.5与Magicoder在代码审查自动化中的微调和提示策略,实验表明微调配合零样本性能最佳,数据不足时宜采用无角色设定的少样本学习。

摘要:

背景:大语言模型(LLMs)的快速发展引发了人们对利用其能力实现代码审查流程自动化的浓厚兴趣。以往研究通常侧重于开发用于代码审查自动化的LLM,但这需要昂贵的资源,对于预算和资源有限的组织而言并不可行。因此,微调和提示工程是利用LLM进行代码审查自动化的两种常用方法。目标:我们旨在基于两种情境研究基于LLM的代码审查自动化的性能,即在通过微调和提示利用LLM时的表现。微调涉及在特定代码审查数据集上训练模型,而提示则涉及提供明确指令以引导模型的生成过程,且不需要特定的代码审查数据集。方法:我们在基于LLM的代码审查自动化中利用模型微调和推理技术(即零样本学习、少样本学习和角色设定)。我们总共研究了两种基于LLM的代码审查自动化(即GPT-3.5和Magicoder)的12种变体,并将它们与Guo等人的方法以及三种现有的代码审查自动化方法进行比较。结果:结合零样本学习对GPT-3.5进行微调,有助于GPT-3.5取得比Guo等人的方法高73.17%–74.23%的EM。此外,当GPT-3.5未经过微调时,采用少样本学习的GPT-3.5比采用零样本学习的GPT-3.5取得高46.38%–659.09%的EM。结论:基于我们的结果,我们建议:(1)用于代码审查自动化的LLM应进行微调以获得最高性能;(2)当数据不足以进行模型微调时(例如冷启动问题),用于代码审查自动化的LLM应采用不带角色设定的少样本学习。

英文摘要:

Context: The rapid evolution of Large Language Models (LLMs) has sparked significant interest in leveraging their capabilities for automating code review processes. Prior studies often focus on developing LLMs for code review automation, yet require expensive resources, which is infeasible for organizations with limited budgets and resources. Thus, fine-tuning and prompt engineering are the two common approaches to leveraging LLMs for code review automation. Objective: We aim to investigate the performance of LLMs-based code review automation based on two contexts, i.e., when LLMs are leveraged by fine-tuning and prompting. Fine-tuning involves training the model on a specific code review dataset, while prompting involves providing explicit instructions to guide the model's generation process without requiring a specific code review dataset. Method: We leverage model fine-tuning and inference techniques (i.e., zero-shot learning, few-shot learning and persona) on LLMs-based code review automation. In total, we investigate 12 variations of two LLMs-based code review automation (i.e., GPT- 3.5 and Magicoder), and compare them with the Guo et al.'s approach and three existing code review automation approaches. Results: The fine-tuning of GPT 3.5 with zero-shot learning helps GPT-3.5 to achieve 73.17% -74.23% higher EM than the Guo et al.'s approach. In addition, when GPT-3.5 is not fine-tuned, GPT-3.5 with few-shot learning achieves 46.38% - 659.09% higher EM than GPT-3.5 with zero-shot learning. Conclusions: Based on our results, we recommend that (1) LLMs for code review automation should be fine-tuned to achieve the highest performance; and (2) when data is not sufficient for model fine-tuning (e.g., a cold-start problem), few-shot learning without a persona should be used for LLMs for code review automation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑