Lyra:一种用于土耳其烤肉式代码生成的基准
Lyra: A Benchmark for Turducken-Style Code Generation
AI总结:
本文提出首个土耳其烤肉式代码生成任务及基准Lyra(Python嵌入SQL),实验表明GPT风格模型最佳,AST精确匹配准确率约24-25.5%,为代码生成带来新挑战。
摘要:
近年来,神经技术已被用于自动生成源代码。虽然这些方法在声明式语言上表现良好,但在命令式语言的数据集上性能较差。由于在现实世界的软件开发中,声明式语言通常嵌入在命令式语言中(即土耳其烤肉式编程),因此声明式语言上的良好结果很难显著减少手动软件开发工作。本文定义了一个新的代码生成任务:给定一个自然语言注释,该任务旨在生成一个带有嵌入声明式语言的基础命令式语言的程序。据我们所知,这是首个土耳其烤肉式代码生成任务。为此,我们提出了Lyra:一个包含嵌入SQL的Python数据集。该数据集包含来自真实项目的2,000个精心标注的数据库操作程序。每个程序都配有一个中文注释和一个英文注释。在我们的实验中,我们采用了Transformer、BERT风格和GPT风格的模型作为基线。在最佳设置下,GPT风格模型的生成性能优于其他模型,其中使用中文和英文注释时的AST精确匹配准确率分别为24%和25.5%。因此,我们认为Lyra为代码生成提供了新的挑战。然而,克服这一挑战可能会显著提升代码生成技术在现实世界软件开发中的适用性。
英文摘要:
Recently, neural techniques have been used to generate source code automatically. While promising for declarative languages, these approaches achieve much poorer performance on datasets for imperative languages. Since a declarative language is typically embedded in an imperative language (i.e., the turducken-style programming) in real-world software development, the promising results on declarative languages can hardly lead to significant reduction of manual software development efforts. In this paper, we define a new code generation task: given a natural language comment, this task aims to generate a program in a base imperative language with an embedded declarative language. To our knowledge, this is the first turducken-style code generation task. For this task, we present Lyra: a dataset in Python with embedded SQL. This dataset contains 2,000 carefully annotated database manipulation programs from real-world projects. Each program is paired with both a Chinese comment and an English comment. In our experiment, we adopted Transformer, BERT-style, and GPT-style models as baselines. In the best setting, the generation performance of GPT-style models is better than others, where the AST exact matching accuracy is 24% and 25.5% when using Chinese and English comments, respectively. Therefore, we believe that Lyra provides a new challenge for code generation. Yet, overcoming this challenge may significantly boost the applicability of code generation techniques for real-world software development.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。