教授代码大语言模型在仓库级代码生成中使用自动补全工具
Teaching Code LLMs to Use Autocompletion Tools in Repository-Level Code Generation
AI总结:
针对仓库级代码生成中依赖错误问题,提出ToolGen方法,通过插入触发标记并微调代码LLM,在线调用自动补全工具,显著提升依赖覆盖率与静态有效性比率。
摘要:
代码大语言模型(LLM)在仓库级代码生成中面临局限性,因为它们缺乏对仓库级依赖关系(例如用户自定义属性)的感知,导致诸如未定义变量和无成员错误等依赖错误。在本工作中,我们引入了ToolGen,一种将自动补全工具集成到代码LLM生成过程中以解决这些依赖关系的方法。ToolGen包含两个主要阶段:触发插入与模型微调(离线阶段),以及工具集成代码生成(在线阶段)。在离线阶段,ToolGen使用特殊标记令牌增强给定代码语料库中的函数,指示触发自动补全工具的位置。这些增强后的函数及其对应的文档字符串随后用于微调选定的代码LLM。在线阶段,ToolGen通过使用微调后的LLM逐步预测令牌来迭代生成函数。每当遇到标记令牌时,ToolGen调用自动补全工具来建议代码补全,并选择最合适的一个。我们进行了全面实验以评估ToolGen在仓库级代码生成中的有效性。为促进评估,我们创建了一个包含671个真实世界代码仓库的基准,并引入了两个新的基于依赖的指标:依赖覆盖率和静态有效性比率。结果表明,ToolGen在三个LLM上将依赖覆盖率显著提高了31.4%至39.1%,静态有效性比率提高了44.9%至57.7%,同时在广泛认可的相似度指标(如BLEU-4、CodeBLEU、编辑相似度和精确匹配)上保持竞争力或有所改进。在CoderEval数据集上,ToolGen在CodeT5和CodeLlama上的Pass@1分别实现了40.0%和25.0%的提升。
英文摘要:
Code large language models (LLMs) face limitations in repository-level code generation due to their lack of awareness of repository-level dependencies (e.g., user-defined attributes), resulting in dependency errors such as undefined-variable and no-member errors. In this work, we introduce ToolGen, an approach that integrates autocompletion tools into the code LLM generation process to address these dependencies. ToolGen comprises two main phases: Trigger Insertion and Model Fine-tuning (Offline), and Tool-integrated Code Generation (Online). During the offline phase, ToolGen augments functions within a given code corpus with a special mark token, indicating positions to trigger autocompletion tools. These augmented functions, along with their corresponding docstrings, are then used to fine-tune a selected code LLM. In the online phase, ToolGen iteratively generates functions by predicting tokens step-by-step using the fine-tuned LLM. Whenever a mark token is encountered, ToolGen invokes the autocompletion tool to suggest code completions and selects the most appropriate one. We conduct comprehensive experiments to evaluate ToolGen's effectiveness in repository-level code generation. To facilitate this evaluation, we create a benchmark comprising 671 real-world code repositories and introduce two new dependency-based metrics: Dependency Coverage and Static Validity Rate. The results demonstrate that ToolGen significantly improves Dependency Coverage by 31.4% to 39.1% and Static Validity Rate by 44.9% to 57.7% across the three LLMs, while maintaining competitive or improved performance in widely recognized similarity metrics such as BLEU-4, CodeBLEU, Edit Similarity, and Exact Match. On the CoderEval dataset, ToolGen achieves improvements of 40.0% and 25.0% in Pass@1 for CodeT5 and CodeLlama, respectively.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。