arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.26480cs.SE
收藏

基于开源大语言模型的自动化提取方法重构:一项对比研究

Automated Extract Method Refactoring with Open-Source LLMs: A Comparative Study

University of Groningen(格罗宁根大学)

Sivajeet Chand, Melih Kilic, Roland Würsching, Sushant Kumar Pandey, Alexander Pretschner

AI总结:

本研究对比评估了5款3B-8B参数的开源LLM在Python代码提取方法重构任务中的表现,发现递归批评改进(RCI)提示策略效果优于单样本提示,最优模型测试通过率超0.8,开发者接受度超70%,并发布了开源基准。

摘要:

尽管提取方法重构(Extract Method refactoring, EMR)在提升代码可读性与可维护性方面具有重要意义,但其自动化实现仍具挑战性,目前很大程度上依赖人工操作。近期开源、资源高效的大语言模型(Large Language Models, LLMs)的发展,为这类高层任务的自动化提供了颇具前景的新途径。本研究针对Python代码的EMR任务,严格评估了5款参数规模在30亿到80亿之间的前沿开源LLM。我们使用自动化指标系统评估功能正确性与代码质量,并通过对比单样本提示(one-shot prompting)与递归批评改进(Recursive criticism and improvement, RCI)方法,探究提示策略的影响。基于RCI的提示在测试通过率与重构质量上始终优于单样本提示。表现最佳的模型Deepseek-Coder-RCI与Qwen2.5-Coder-RCI,测试通过率(test pass percentage, TPP)分别达到0.829和0.808,同时将单方法代码行数(lines of code, LOC)从12.103分别降至6.192和5.577,圈复杂度(cyclomatic complexity, CC)从4.602分别降至3.453和3.294。针对RCI生成的重构结果开展的开发者调研显示,其接受度超过70%,其中Qwen2.5-Coder在所有评估维度中评分最高。相比之下,原始代码的评分低于中性水平,尤其在可读性与可维护性方面,凸显了由高质量提示引导的自动化重构的优势。尽管CC、LOC等传统指标能提供有用的参考信号,但它们常与人类判断存在偏差,这强调了人在回路评估的必要性。我们的开源基准为未来基于LLM的自动化重构研究奠定了基础。

英文摘要:

Automating the Extract Method refactoring (EMR) remains challenging and largely manual despite its importance in improving code readability and maintainability. Recent advances in open-source, resource-efficient Large Language Models (LLMs) offer promising new approaches for automating such high-level tasks. In this work, we critically evaluate five state-of-the-art open-source LLMs, spanning 3B to 8B parameter sizes, on the EMR task for Python code. We systematically assess functional correctness and code quality using automated metrics and investigate the impact of prompting strategies by comparing one-shot prompting to a Recursive criticism and improvement (RCI) approach. RCI-based prompting consistently outperforms one-shot prompting in test pass rates and refactoring quality. The best-performing models, Deepseek-Coder-RCI and Qwen2.5-Coder-RCI, achieve test pass percentage (TPP) scores of 0.829 and 0.808, while reducing lines of code (LOC) per method from 12.103 to 6.192 and 5.577, and cyclomatic complexity (CC) from 4.602 to 3.453 and 3.294, respectively. A developer survey on RCI-generated refactorings shows over 70% acceptance, with Qwen2.5-Coder rated highest across all evaluation criteria. In contrast, the original code scored below neutral, particularly in readability and maintainability, underscoring the benefits of automated refactoring guided by quality prompts. While traditional metrics like CC and LOC provide useful signals, they often diverge from human judgments, emphasizing the need for human-in-the-loop evaluation. Our open-source benchmark offers a foundation for future research on automated refactoring with LLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑