CoderEval:基于生成式预训练模型的实用代码生成基准
CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Models
AI总结:
本文提出 CoderEval 基准,包含 230 个 Python 和 230 个 Java 真实项目代码生成任务及自动执行平台,用于评估模型在非独立函数和六级上下文依赖下的实用代码生成能力,并发现现有模型生成独立函数的效果显著优于非独立函数。
摘要:
基于预训练和微调范式的代码生成模型已越来越多地被学术界和工业界尝试,产生了诸如 Codex、CodeGen 和 PanGu-Coder 等知名工业模型。为了评估这些模型的有效性,现有多个基准被提出,但它们只包含生成独立函数(standalone function)的用例,即该函数只能调用或访问内置函数和标准库。然而,非独立函数(non-standalone functions)通常未被现有基准覆盖,却在流行开源项目中占函数总数的 70% 以上;仅在独立函数上评估模型有效性无法反映这些模型在实用代码生成场景中的有效性。为了弥合上述差距,本文提出了一个名为 CoderEval 的基准,包含从流行的真实世界开源项目中精心整理的 230 个 Python 和 230 个 Java 代码生成任务,并提供一个自包含的执行平台来自动评估生成代码的功能正确性。CoderEval 支持来自六个上下文依赖级别的代码生成任务,其中上下文指的是在被生成函数外部、但位于所依赖的第三方库、当前类、文件或项目内部定义的代码元素,例如类型、API、变量和常量。CoderEval 可用于评估模型在生成超越独立函数的代码时的有效性。通过在 CoderEval 上评估三个代码生成模型,我们发现这些模型在生成独立函数方面的有效性显著高于生成非独立函数。我们的分析突显了当前进展,并指出了未来通过利用上下文信息来进一步提高模型在实用代码生成中有效性的方向。
英文摘要:
Code generation models based on the pre-training and fine-tuning paradigm have been increasingly attempted by both academia and industry, resulting in well-known industrial models such as Codex, CodeGen, and PanGu-Coder. To evaluate the effectiveness of these models, multiple existing benchmarks are proposed, including only cases of generating a standalone function, i.e., a function that may invoke or access only built-in functions and standard libraries. However, non-standalone functions, which typically are not included in the existing benchmarks, constitute more than 70% of the functions in popular open-source projects, and evaluating models' effectiveness on standalone functions cannot reflect these models' effectiveness on pragmatic code generation scenarios. To help bridge the preceding gap, in this paper, we propose a benchmark named CoderEval, consisting of 230 Python and 230 Java code generation tasks carefully curated from popular real-world open-source projects and a self-contained execution platform to automatically assess the functional correctness of generated code. CoderEval supports code generation tasks from six levels of context dependency, where context refers to code elements such as types, APIs, variables, and consts defined outside the function under generation but within the dependent third-party libraries, current class, file, or project. CoderEval can be used to evaluate the effectiveness of models in generating code beyond only standalone functions. By evaluating three code generation models on CoderEval, we find that the effectiveness of these models in generating standalone functions is substantially higher than that in generating non-standalone functions. Our analysis highlights the current progress and pinpoints future directions to further improve a model's effectiveness by leveraging contextual information for pragmatic code generation.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。