arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.03029cs.SEcs.AI
收藏

调查 LLM 生成代码中的代码坏味道

Investigating The Smells of LLM Generated Code

School of Engineering, Computing and Mathematics, Oxford Brookes University(工程、计算与数学学院,奥克斯福德布鲁克斯大学)

Debalina Ghosh Paul, Hong Zhu, Ian Bayley

AI总结:

提出基于场景的代码坏味道评估方法,利用自动化系统比较四款 LLM 生成的 Java 程序与专业参考代码,发现其代码质量普遍较差且随任务复杂度上升。

摘要:

背景:大型语言模型(LLM)正越来越多地用于生成程序代码。已有大量研究关注生成代码的功能正确性,但关于代码质量的研究却少得多。目标:在本研究中,我们提出一种基于场景的方法来评估 LLM 生成代码的质量,以识别应提升 LLM 生成代码质量的最薄弱场景。方法:该方法测量代码坏味道(code smells)这一代码质量的重要指标,并将其与由专业编写代码的参考解决方案构成的基线进行比较。测试数据集根据代码主题和编程任务复杂度划分为多个子集,以表示使用 LLM 进行代码生成的不同场景。我们还将介绍为此目的构建的自动化测试系统,并报告针对向四个最先进 LLM(Gemini Pro、ChatGPT、Codex 和 Falcon)给出提示后所生成 Java 程序的实验。结果:我们发现,与参考解决方案相比,LLM 生成代码的代码坏味道发生率更高。Falcon 表现得相对最好,坏味道增加 42.28%,其次是 Gemini Pro(62.07%)、ChatGPT(65.05%),最后是 Codex(84.97%)。所有 LLM 的平均坏味道增加为 63.34%,其中实现坏味道增加 73.35%,设计坏味道增加 21.42%。我们还发现,对于更复杂的编程任务和更高级的主题(例如涉及面向对象概念的任务和主题),代码坏味道的增加更为明显。结论:就代码坏味道而言,LLM 在不同编程任务复杂度和主题上的表现,与相应场景下人工编写代码的质量高度相关。然而,LLM 生成代码的质量明显差于人工编写代码。

英文摘要:

Context: Large Language Models (LLMs) are increasingly being used to generate program code. Much research has been reported on the functional correctness of generated code, but there is far less on code quality. Objectives: In this study, we propose a scenario-based method of evaluating the quality of LLM-generated code to identify the weakest scenarios, for which the quality of LLM-generated code should be improved. Methods: The method measures code smells, an important indicator of code quality, and compares them with a baseline formed from reference solutions of professionally written code. The test dataset is divided into various subsets according to the topics of the code and complexity of the coding tasks to represent different scenarios of using LLMs for code generation. We will also present an automated test system for this purpose and report experiments with the Java programs generated in response to prompts given to four state-of-the-art LLMs: Gemini Pro, ChatGPT, Codex, and Falcon. Results: We find that LLM-generated code has a higher incidence of code smells compared to reference solutions. Falcon performed the least badly, with a smell increase of 42.28%, followed by Gemini Pro (62.07%), ChatGPT (65.05%) and finally Codex (84.97%). The average smell increase across all LLMs was 63.34%, comprising 73.35% for implementation smells and 21.42% for design smells. We also found that the increase in code smells is greater for more complex coding tasks and for more advanced topics, such as those involving object-orientated concepts. Conclusion: In terms of code smells, LLMs' performances on various coding task complexities and topics are highly correlated to the quality of human-written code in the corresponding scenarios. However, the quality of LLM-generated code is noticeably poorer than human-written code.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑