arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2206.07585cs.PLcs.AIcs.LGcs.SE
收藏

NatGen:通过“自然化”source code进行生成式预训练

NatGen: Generative pre-training by "Naturalizing" source code

Columbia University(哥伦比亚大学);University of California, Davis(加州大学戴维斯分校)

Saikat Chakraborty, Toufique Ahmed, Yangruibo Ding, Premkumar Devanbu, Baishakhi Ray

摘要:

在过去的几年里,用于源代码的预训练生成语言模型(例如PLBART,CodeT 5,SPT-Code)在几项任务上取得了很好的成绩,包括代码生成和翻译。这些模型采用了不同的预训练目标,以自我监督的方式从非常大规模的语料库中学习代码构建的统计数据;预训练模型的成功在很大程度上取决于这些预训练目标。本文提出了一个新的预训练目标,“自然化”的源代码,利用代码的双峰,双通道(正式和自然通道)的性质。与自然语言不同,代码的双峰、双通道性质允许我们大规模地生成语义等价的代码。我们引入了六类语义保持转换来引入不自然的代码形式,然后迫使我们的模型产生更自然的开发人员编写的原始程序。在没有明确的人工监督的情况下,学习在大型开源代码语料库中大规模生成等效但更自然的代码,有助于模型学习摄取和生成代码。我们在三个生成软件工程任务中微调我们的模型:代码生成,代码翻译和代码细化,使用有限的人工策划的标记数据,并实现最先进的性能媲美CodeT 5。我们表明,我们的预训练模型在zero-shot和少样本学习方面特别有竞争力,并且在学习代码属性(例如,语法、数据流)。

英文摘要:

Pre-trained Generative Language models (e.g. PLBART, CodeT5, SPT-Code) for source code yielded strong results on several tasks in the past few years, including code generation and translation. These models have adopted varying pre-training objectives to learn statistics of code construction from very large-scale corpora in a self-supervised fashion; the success of pre-trained models largely hinges on these pre-training objectives. This paper proposes a new pre-training objective, "Naturalizing" of source code, exploiting code's bimodal, dual-channel (formal & natural channels) nature. Unlike natural language, code's bimodal, dual-channel nature allows us to generate semantically equivalent code at scale. We introduce six classes of semantic preserving transformations to introduce un-natural forms of code, and then force our model to produce more natural original programs written by developers. Learning to generate equivalent, but more natural code, at scale, over large corpora of open-source code, without explicit manual supervision, helps the model learn to both ingest & generate code. We fine-tune our model in three generative Software Engineering tasks: code generation, code translation, and code refinement with limited human-curated labeled data and achieve state-of-the-art performance rivaling CodeT5. We show that our pre-trained model is especially competitive at zero-shot and few-shot learning, and better at learning code properties (e.g., syntax, data flow).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑