arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.05926cs.SE
收藏

使用大语言模型生成提交信息:一项初步研究

Using Large Language Models for Commit Message Generation: A Preliminary Study

Linghao Zhang, Jingshu Zhao, Chong Wang, Peng Liang

AI总结:

本文通过实验评估Llama 2和ChatGPT在提交信息生成中的表现,发现大语言模型在人工评估中显著优于现有方法,并揭示了常用自动评估指标的局限性。

摘要:

提交信息是对一次提交中代码变更的文本描述,是Git版本控制系统(VCS)的关键组成部分。它捕捉了软件更新的本质。因此,它可以帮助开发者理解代码演化,并促进开发者之间的高效协作。然而,编写良好且有价值的提交信息既耗时又费力。一些研究者已经对提交信息的自动生成开展了广泛研究,并为此提出了若干方法,例如基于生成的模型和基于检索的模型。然而,很少有研究探讨大语言模型(LLMs)是否能够被用于自动且有效地生成提交信息。为此,本文设计并进行了一系列实验,以全面评估流行的开源和闭源大语言模型,即Llama 2和ChatGPT,在提交信息生成中的性能。结果表明,在考虑BLEU和Rouge-L指标时,大语言模型在某些指标上超越了现有方法,但在其他指标上落后。然而,经过人工评估后,大语言模型相对于所有这些现有方法表现出明显优势。尤其是在366个样本中,78%的样本里由大语言模型生成的提交信息被人类评估为最佳。这项工作不仅揭示了使用大语言模型生成提交信息的巨大潜力,还探讨了常用指标在评估自动生成提交信息质量方面的局限性。

英文摘要:

A commit message is a textual description of the code changes in a commit, which is a key part of the Git version control system (VCS). It captures the essence of software updating. Therefore, it can help developers understand code evolution and facilitate efficient collaboration between developers. However, it is time-consuming and labor-intensive to write good and valuable commit messages. Some researchers have conducted extensive studies on the automatic generation of commit messages and proposed several methods for this purpose, such as generationbased and retrieval-based models. However, seldom studies explored whether large language models (LLMs) can be used to generate commit messages automatically and effectively. To this end, this paper designed and conducted a series of experiments to comprehensively evaluate the performance of popular open-source and closed-source LLMs, i.e., Llama 2 and ChatGPT, in commit message generation. The results indicate that considering the BLEU and Rouge-L metrics, LLMs surpass the existing methods in certain indicators but lag behind in others. After human evaluations, however, LLMs show a distinct advantage over all these existing methods. Especially, in 78% of the 366 samples, the commit messages generated by LLMs were evaluated by humans as the best. This work not only reveals the promising potential of using LLMs to generate commit messages, but also explores the limitations of commonly used metrics in evaluating the quality of auto-generated commit messages.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑