arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2411.04444cs.SE
收藏

大型语言模型在自动化软件重构中潜力的实证研究

An Empirical Study on the Potential of LLMs in Automated Software Refactoring

Beijing Institute of Technology(北京理工大学);Peking University(北京大学);University of Cincinnati(辛辛那提大学);National Innovation Institute of Defense Technology(国防科技大学创新研究院)

Bo Liu, Yanjie Jiang, Yuxia Zhang, Nan Niu, Guangjie Li, Hui Liu

AI总结:

本研究通过实证分析发现LLM在重构机会识别上表现有限,但通过提示优化可大幅提升;同时提出RefactoringMirror策略以规避不安全重构,兼顾LLM智能与代码安全。

摘要:

大型语言模型(LLMs)的最新进展使得使用LLMs自动重构源代码成为可能。然而,与人类专家相比,LLMs在自动且准确地进行重构方面的表现如何仍不清楚。为填补这一空白,本文进行了一项实证研究,以调查LLMs在自动化软件重构中的潜力,重点关注重构机会的识别和重构解决方案的推荐。我们首先构建了一个高质量的重构数据集,包含来自20个项目的180个真实重构案例,并在此数据集上进行实证研究。以待重构的Java文档作为输入,ChatGPT和Gemini分别仅识别出180个重构机会中的28个和7个。然而,在提示中解释预期的重构子类别并缩小搜索空间,将ChatGPT的成功率从15.6%大幅提升至86.7%。关于重构解决方案的推荐,ChatGPT为180个重构推荐了176个解决方案,其中63.6%的推荐方案与人类专家构建的方案相当(甚至更好)。然而,ChatGPT建议的176个解决方案中有13个以及Gemini建议的137个解决方案中有9个是不安全的,因为它们要么改变了源代码的功能,要么引入了语法错误,这表明了基于LLM的重构存在风险。为此,我们提出了一种名为RefactoringMirror的检测-重新应用策略,以避免此类不安全的重构。通过使用经过充分测试的重构引擎将识别出的重构重新应用于原始代码,我们可以有效减轻与基于LLM的自动化重构相关的风险,同时仍然利用LLM的智能来获得有价值的重构建议。

英文摘要:

Recent advances in large language models (LLMs), make it potentially feasible to automatically refactor source code with LLMs. However, it remains unclear how well LLMs perform compared to human experts in conducting refactorings automatically and accurately. To fill this gap, in this paper, we conduct an empirical study to investigate the potential of LLMs in automated software refactoring, focusing on the identification of refactoring opportunities and the recommendation of refactoring solutions. We first construct a high-quality refactoring dataset comprising 180 real-world refactorings from 20 projects, and conduct the empirical study on the dataset. With the to-be-refactored Java documents as input, ChatGPT and Gemini identified only 28 and 7 respectively out of the 180 refactoring opportunities. However, explaining the expected refactoring subcategories and narrowing the search space in the prompts substantially increased the success rate of ChatGPT from 15.6% to 86.7%. Concerning the recommendation of refactoring solutions, ChatGPT recommended 176 refactoring solutions for the 180 refactorings, and 63.6% of the recommended solutions were comparable to (even better than) those constructed by human experts. However, 13 out of the 176 solutions suggested by ChatGPT and 9 out of the 137 solutions suggested by Gemini were unsafe in that they either changed the functionality of the source code or introduced syntax errors, which indicate the risk of LLM-based refactoring. To this end, we propose a detect-and-reapply tactic, called RefactoringMirror, to avoid such unsafe refactorings. By reapplying the identified refactorings to the original code using thoroughly tested refactoring engines, we can effectively mitigate the risks associated with LLM-based automated refactoring while still leveraging LLM's intelligence to obtain valuable refactoring recommendations.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑