VersiCode:迈向版本可控的代码生成
VersiCode: Towards Version-controllable Code Generation
AI总结:
本文提出版本特定代码补全(VSCC)和版本感知代码迁移(VACM)两项新任务,构建综合Python数据集VersiCode及新评估指标CDC@1,揭示即使GPT-4o等前沿模型在版本可控代码生成上也面临重大挑战。
摘要:
大型语言模型(LLMs)在代码生成方面取得了巨大进展,但现有研究未能考虑到以频繁库更新为特征的软件开发动态本质。这一差距显著限制了LLMs在真实场景中的部署。本文提出了两个旨在弥合这一差距的新颖任务:版本特定代码补全(VSCC)和版本感知代码迁移(VACM)。同时,我们引入了VersiCode,这是一个专门设计用于在这两个任务上评估LLMs的综合Python数据集,以及一种新颖的评估指标,关键差异检查(CDC@1),用于根据不断演变的API要求评估代码生成。我们在VersiCode上进行了广泛评估,结果显示版本可控的代码生成确实是一个重大挑战,即使对于GPT-4o和其他强大的前沿模型也是如此。我们相信,这些新颖的任务、数据集和指标开辟了一个新的重要研究方向,将进一步增强LLMs在现实世界中的适用性。代码和资源可在https://github.com/wutong8023/VersiCode获取。
英文摘要:
Large Language Models (LLMs) have made tremendous strides in code generation, but existing research fails to account for the dynamic nature of software development, marked by frequent library updates. This gap significantly limits LLMs' deployment in realistic settings. In this paper, we propose two novel tasks aimed at bridging this gap: version-specific code completion (VSCC) and version-aware code migration (VACM). In conjunction, we introduce VersiCode, a comprehensive Python dataset specifically designed to evaluate LLMs on these two tasks, together with a novel evaluation metric, Critical Diff Check (CDC@1), which assesses code generation against evolving API requirements. We conduct an extensive evaluation on VersiCode, which reveals that version-controllable code generation is indeed a significant challenge, even for GPT-4o and other strong frontier models. We believe the novel tasks, dataset, and metric open up a new, important research direction that will further enhance LLMs' real-world applicability. The code and resources can be found at https://github.com/wutong8023/VersiCode.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。