arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.07959cs.SEcs.AI
收藏

大语言模型时代的源代码摘要生成

Source Code Summarization in the Era of Large Language Models

College of Computing and Data Science, Nanyang Technological University Singapore(计算与数据科学学院,新加坡南洋理工大学);State Key Laboratory for Novel Software Technology, Nanjing University(新型软件技术国家重点实验室,南京大学);School of Computer Science and Engineering, University of New South Wales(计算机科学与工程学院,新南威尔士大学);School of Big Data and Software Engineering, Chongqing University(大数据与软件工程学院,重庆大学)

Weisong Sun, Yun Miao, Yuekang Li, Hongyu Zhang, Chunrong Fang, Yi Liu, Gelei Deng, Yang Liu, Zhenyu Chen

AI总结:

本文针对大语言模型时代的代码摘要展开系统研究,分析自动评估方法、提示技术、模型设置及编程语言类型的影响,发现GPT-4评估与人类评估最接近、零样本提示可能优于高级提示等结论,还意外发现7B参数的CodeLlama-Instruct在特定摘要任务上优于GPT-4。

摘要:

为支持软件开发人员理解和维护程序,已提出多种自动(源)代码摘要技术,为给定代码片段生成简洁的自然语言摘要(即注释)。近年来,大语言模型(LLMs)的出现极大提升了代码相关任务的性能。本文对LLMs时代的代码摘要进行了系统且全面的研究,涵盖LLM-based代码摘要工作流中的多个方面。具体而言,我们首先考察了评估LLMs生成摘要质量的主流自动评估方法,发现GPT-4评估方法的结果与人类评估最接近。然后,我们探究了五种提示技术(零样本、少样本、思维链、批判和专家提示)在使LLMs适应代码摘要任务中的有效性,结果与预期相反,高级提示技术可能并不优于简单的零样本提示。接下来,我们研究了LLMs的模型设置(包括top_p和温度参数)对生成摘要质量的影响,发现这两个参数对摘要质量的影响因基础LLM和编程语言而异,但影响模式相似。此外,我们调查了LLMs对不同类型编程语言代码片段的摘要能力,结果显示,与其他语言类型相比,LLMs对逻辑编程语言代码的摘要性能欠佳。最后,我们意外发现,7B参数的CodeLlama-Instruct在生成描述代码实现细节和断言代码属性的摘要方面,性能优于先进的GPT-4。我们希望我们的研究结果能为LLMs时代的代码摘要提供全面的理解。

英文摘要:

To support software developers in understanding and maintaining programs, various automatic (source) code summarization techniques have been proposed to generate a concise natural language summary (i.e., comment) for a given code snippet. Recently, the emergence of large language models (LLMs) has led to a great boost in the performance of code-related tasks. In this paper, we undertake a systematic and comprehensive study on code summarization in the era of LLMs, which covers multiple aspects involved in the workflow of LLM-based code summarization. Specifically, we begin by examining prevalent automated evaluation methods for assessing the quality of summaries generated by LLMs and find that the results of the GPT-4 evaluation method are most closely aligned with human evaluation. Then, we explore the effectiveness of five prompting techniques (zero-shot, few-shot, chain-of-thought, critique, and expert) in adapting LLMs to code summarization tasks. Contrary to expectations, advanced prompting techniques may not outperform simple zero-shot prompting. Next, we investigate the impact of LLMs' model settings (including top\_p and temperature parameters) on the quality of generated summaries. We find the impact of the two parameters on summary quality varies by the base LLM and programming language, but their impacts are similar. Moreover, we canvass LLMs' abilities to summarize code snippets in distinct types of programming languages. The results reveal that LLMs perform suboptimally when summarizing code written in logic programming languages compared to other language types. Finally, we unexpectedly find that CodeLlama-Instruct with 7B parameters can outperform advanced GPT-4 in generating summaries describing code implementation details and asserting code properties. We hope that our findings can provide a comprehensive understanding of code summarization in the era of LLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑