arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.02824cs.SE
收藏

探索LLM在代码变更相关任务上的能力

Exploring the Capabilities of LLMs for Code Change Related Tasks

Lishui Fan, Jiakun Liu, Zhongxin Liu, David Lo, Xin Xia, Shanping Li

AI总结:

本文实证评估超10亿参数LLM在代码审查、提交信息生成和即时注释更新上的ICL与PEFT表现,揭示示例、模型规模及变更类型对性能的影响。

摘要:

开发者每天都要处理与代码变更相关的任务,例如代码审查。面向预训练代码和代码变更的模型已被调整用于帮助开发者完成此类任务。近来,大型语言模型(LLM)已在代码相关任务中展现出有效性。然而,现有的代码LLM关注的是通用代码语法和语义,而非两个代码版本之间的差异。因此,LLM在代码变更相关任务上表现如何仍是一个开放问题。为回答该问题,我们使用参数量超过10亿的LLM,在三项代码变更相关任务(即代码审查生成、提交信息生成和即时注释更新)上,结合上下文学习(ICL)与参数高效微调(PEFT,包括LoRA和prefix-tuning)开展实证研究。我们观察到,在没有示例时LLM性能较差,提供示例通常会改善性能,但更多示例并不总能带来更好性能。经LoRA微调的LLM可达到与最先进的小型预训练模型相当的性能。更大的模型并不总是更好,但Llama 2和Code Llama系列始终表现最佳。最佳LLM在仅修改注释的代码变更上优于小型预训练模型,并在其他代码变更上表现相当。我们建议未来工作应更多关注引导LLM学习与代码变更(而非注释)相关的特有知识,以完成代码变更相关任务。

英文摘要:

Developers deal with code-change-related tasks daily, e.g., reviewing code. Pre-trained code and code-change-oriented models have been adapted to help developers with such tasks. Recently, large language models (LLMs) have shown their effectiveness in code-related tasks. However, existing LLMs for code focus on general code syntax and semantics rather than the differences between two code versions. Thus, it is an open question how LLMs perform on code-change-related tasks. To answer this question, we conduct an empirical study using \textgreater 1B parameters LLMs on three code-change-related tasks, i.e., code review generation, commit message generation, and just-in-time comment update, with in-context learning (ICL) and parameter-efficient fine-tuning (PEFT, including LoRA and prefix-tuning). We observe that the performance of LLMs is poor without examples and generally improves with examples, but more examples do not always lead to better performance. LLMs tuned with LoRA have comparable performance to the state-of-the-art small pre-trained models. Larger models are not always better, but \textsc{Llama~2} and \textsc{Code~Llama} families are always the best. The best LLMs outperform small pre-trained models on the code changes that only modify comments and perform comparably on other code changes. We suggest future work should focus more on guiding LLMs to learn the knowledge specific to the changes related to code rather than comments for code-change-related tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑