School of Engineering, Computing and Mathematics, Oxford Brookes University(工程、计算与数学学院,奥克斯福德布鲁克斯大学)
Debalina Ghosh Paul, Hong Zhu, Ian Bayley
备注:
This is a revised version of the paper done in June 2026. It improves the presentation of the work and the statistical analysis of the data. The raw data of the experiments are the same as the previous version
Context: Large Language Models (LLMs) are increasingly being used to generate program code. Much research has been reported on the functional correctness of generated code, but there is far less on code quality. Objectives: In this study, we propose a scenario-based method of evaluating the quality of LLM-generated code to identify the weakest scenarios, for which the quality of LLM-generated code should be improved. Methods: The method measures code smells, an important indicator of code quality, and compares them with a baseline formed from reference solutions of professionally written code. The test dataset is divided into various subsets according to the topics of the code and complexity of the coding tasks to represent different scenarios of using LLMs for code generation. We will also present an automated test system for this purpose and report experiments with the Java programs generated in response to prompts given to four state-of-the-art LLMs: Gemini Pro, ChatGPT, Codex, and Falcon. Results: We find that LLM-generated code has a higher incidence of code smells compared to reference solutions. Falcon performed the least badly, with a smell increase of 42.28%, followed by Gemini Pro (62.07%), ChatGPT (65.05%) and finally Codex (84.97%). The average smell increase across all LLMs was 63.34%, comprising 73.35% for implementation smells and 21.42% for design smells. We also found that the increase in code smells is greater for more complex coding tasks and for more advanced topics, such as those involving object-orientated concepts. Conclusion: In terms of code smells, LLMs' performances on various coding task complexities and topics are highly correlated to the quality of human-written code in the corresponding scenarios. However, the quality of LLM-generated code is noticeably poorer than human-written code.
On the Impact of Requirement Smells in LLM-Based Code Generation
fortiss GmbH(fortiss有限公司);Netlight Consulting(Netlight咨询公司);Technical University of Munich(慕尼黑工业大学);Chalmers University of Technology(查尔姆斯理工大学);Trinity College Dublin(都柏林三一学院);University of Duisburg-Essen(杜伊斯堡-埃森大学);Blekinge Institute of Technology(布莱金厄理工学院)
ShizhenGPT: Towards Multimodal LLMs for Traditional Chinese Medicine
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Freedom AI;National Health Data Institute, Shenzhen(深圳国家健康数据研究院);Shenzhen Research Institute of Big Data(深圳大数据研究院);Guangdong University of Technology(广东工业大学)
Investigating More Explainable and Partition-Free Compositionality Estimation for LLMs: A Rule-Generation Perspective
National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University(国家多媒体信息处理重点实验室,计算机学院,北京大学);OPPO AI Center(OPPO人工智能中心)
On the Diffusion of Test Smells in LLM-Generated Unit Tests
University of Luxembourg(卢森堡大学);Nanjing University of Science and Technology(南京理工大学);Bilkent University(比尔肯特大学);Singapore Management University(新加坡管理大学)
Confusion is the Final Barrier: Rethinking Jailbreak Evaluation and Investigating the Real Misuse Threat of LLMs
State Key Lab of Processors, Institute of Computing Technology, CAS(中国科学院计算技术研究所状态关键实验室);University of Chinese Academy of Sciences(中国科学院大学);People’s Public Security University of China(中国人民公安大学);Chinese University of Hong Kong(香港大学)