arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.14007cs.SE
收藏

关于由人工智能生成的源代码注释质量:全面评估

On the Quality of AI-Generated Source Code Comments: A Comprehensive Evaluation

Federal University of Minas Gerais (UFMG)(米纳斯吉拉斯联邦大学);Pontifical Catholic University of Minas Gerais (PUC Minas)(米纳斯吉拉斯天主教 Pontifical 大学)

Ian Guelman, Arthur Gregório Leal, Laerte Xavier, Marco Tulio Valente

AI总结:

本文评估了AI生成的源代码注释质量,发现其与原始注释相当或更优,并揭示了需要新的文档特定度量。

摘要:

本文研究了由大型语言模型(LLMs)自动生成的源代码注释质量。尽管基于AI的注释生成已成为减少开发者文档工作量的有希望的解决方案,但先前的研究受到小数据集或仅依赖传统信息检索(IR)度量的限制,这些度量不足以捕捉文档质量。为了解决这些限制,我们对142个类和273个方法进行了大规模实证研究,这些方法是在评估模型的训练截止后创建的。对于每个代码元素,我们使用三个LLMs(GPT-3.5 Turbo、GPT-4o和DeepSeek-V3)生成Javadoc注释。对注释的定性评估由两位专家独立进行,结果显示58.8%的注释与原始注释相当,27.7%的注释优于原始注释。使用BLEU、ROUGE-L和METEOR进行的定量分析证实,基于IR的度量无法可靠地反映人类评估,揭示了需要新的文档特定度量的必要性。最后,相关性分析表明,代码属性(大小、复杂性和耦合)与注释质量之间存在略微正相关,证实了LLMs从更丰富的上下文信息中受益。

英文摘要:

This paper investigates the quality of source code comments automatically generated by Large Language Models (LLMs). While AI-based comment generation has emerged as a promising solution to reduce developers' documentation effort, prior studies have been limited by small datasets or by relying solely on traditional Information Retrieval (IR) metrics, which are insufficient to capture documentation quality. To address these limitations, we conducted a large-scale empirical study on 142 classes and 273 methods created after the training cut-off of the evaluated models. For each code element, we generated Javadoc comments using three LLMs (GPT-3.5 Turbo, GPT-4o, and DeepSeek-V3). A qualitative assessment of the comments-performed independently by two experts-showed that 58.8% were equivalent to, and 27.7% superior to, the original comments. A quantitative analysis using BLEU, ROUGE-L, and METEOR confirmed that IR-based metrics do not reliably reflect human evaluations, revealing the need for new documentation-specific metrics. Finally, correlation analyses indicated slightly positive relationships between code properties (size, complexity, coupling) and comment quality, confirming that LLMs benefit from richer contextual information.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑