arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2405.01573cs.SEcs.AI
收藏

基于在代码仓库上进行迭代式、工具增强推理的自然语言类级代码生成

Class-Level Code Generation from Natural Language Using Iterative, Tool-Enhanced Reasoning over Repository

Microsoft Research, India(微软研究院(印度))

Ajinkya Deshpande, Anmol Agarwal, Shashank Shet, Arun Iyer, Aditya Kanade, Ramakrishna Bairi, Suresh Parthasarathy

AI总结:

针对LLMs在真实仓库中进行类级代码生成的挑战,提出RepoClassBench基准和基于智能体框架的RRR方法,通过静态分析工具迭代推理仓库上下文,显著提升了跨语言类级代码生成性能。

摘要:

LLMs在代码生成任务中展现了巨大潜力,在各类基准测试的函数或语句级生成上取得了可喜成果。然而,在真实软件仓库背景下创建类等代码制品的复杂性仍未被充分探索。先前研究将类级生成视为孤立任务,忽略了真实软件环境中特有的复杂依赖与交互。为弥补这一空白,我们引入了RepoClassBench,这是一个旨在严格评估LLMs在真实仓库中生成复杂类级代码能力的综合基准。RepoClassBench包含从精选仓库中提取的Java、Python和C#的“自然语言到类生成”任务。我们确保数据集中的每个类不仅具有仓库内的跨文件依赖关系,还包含相应的测试用例以验证其功能。我们发现,当前模型难以应对基准提出的现实挑战,主要原因是它们对相关仓库上下文的接触有限。为解决这一缺陷,我们提出了Retrieve-Repotools-Reflect (RRR),这是一种新方法,在基于智能体的框架中为LLMs配备静态分析工具,以迭代地导航和推理仓库级上下文。实验表明,RRR在RepoClassBench上显著优于现有基线,展示了其在跨编程语言和各种设置下的有效性。我们的发现强调,代码生成基准迫切需要纳入仓库级依赖,以更准确地反映软件开发的复杂性。我们的工作展示了利用专门工具增强LLMs理解仓库上下文的益处。我们计划公开数据集和评估工具。

英文摘要:

LLMs have demonstrated significant potential in code generation tasks, achieving promising results at the function or statement level across various benchmarks. However, the complexities associated with creating code artifacts like classes, particularly within the context of real-world software repositories, remain underexplored. Prior research treats class-level generation as an isolated task, neglecting the intricate dependencies & interactions that characterize real-world software environments. To address this gap, we introduce RepoClassBench, a comprehensive benchmark designed to rigorously evaluate LLMs in generating complex, class-level code within real-world repositories. RepoClassBench includes "Natural Language to Class generation" tasks across Java, Python & C# from a selection of repositories. We ensure that each class in our dataset not only has cross-file dependencies within the repository but also includes corresponding test cases to verify its functionality. We find that current models struggle with the realistic challenges posed by our benchmark, primarily due to their limited exposure to relevant repository contexts. To address this shortcoming, we introduce Retrieve-Repotools-Reflect (RRR), a novel approach that equips LLMs with static analysis tools to iteratively navigate & reason about repository-level context in an agent-based framework. Our experiments demonstrate that RRR significantly outperforms existing baselines on RepoClassBench, showcasing its effectiveness across programming languages & under various settings. Our findings emphasize the critical need for code-generation benchmarks to incorporate repo-level dependencies to more accurately reflect the complexities of software development. Our work shows the benefits of leveraging specialized tools to enhance LLMs' understanding of repository context. We plan to make our dataset & evaluation harness public.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑