arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.17622cs.SE
收藏

大语言模型时代的提交消息

Commit Messages in the Age of Large Language Models

Cristina V. Lopes, Vanessa I. Klotzman, Iris Ma, Iftekar Ahmed

AI总结:

本研究评估ChatGPT基于代码变更生成提交消息的性能,发现其效果远超专门训练的现有ACMG方法,生成消息准确高质,同时对失效案例进行了分类分析。

摘要:

提交消息是对代码库所做变更的说明,存储在版本控制系统中,可帮助开发者在代码库演进过程中理解其内容。然而,编写提交消息既繁琐,不同开发者的输出也不一致。为解决这一问题,研究者尝试了多种自动生成提交消息的方法,包括基于规则、基于检索和基于学习的方法。大语言模型的发展为提交消息生成提供了新的可能。本研究评估了OpenAI的ChatGPT基于代码变更生成提交消息的性能,将其结果与专门在提交数据上训练的现有自动提交消息生成方法进行对比,旨在评估大型预训练语言模型生成的提交消息在定量和定性层面的可接受程度。研究发现,ChatGPT的性能比现有自动提交消息生成(ACMG)方法高出数个数量级,且其生成的消息通常既准确又高质量。此外,本文还对ChatGPT失效的案例进行了分类并提供了相关洞见。

英文摘要:

Commit messages are explanations of changes made to a codebase that are stored in version control systems. They help developers understand the codebase as it evolves. However, writing commit messages can be tedious and inconsistent among developers. To address this issue, researchers have tried using different methods to automatically generate commit messages, including rule-based, retrieval-based, and learning-based approaches. Advances in large language models offer new possibilities for generating commit messages. In this study, we evaluate the performance of OpenAI's ChatGPT for generating commit messages based on code changes. We compare the results obtained with ChatGPT to previous automatic commit message generation methods that have been trained specifically on commit data. Our goal is to assess the extent to which large pre-trained language models can generate commit messages that are both quantitatively and qualitatively acceptable. We found that ChatGPT was able to outperform previous Automatic Commit Message Generation (ACMG) methods by orders of magnitude, and that, generally, the messages it generates are both accurate and of high-quality. We also provide insights, and a categorization, for the cases where it fails.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑