arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2109.00859cs.CLcs.PL
收藏

CodeT5:用于代码理解和生成的标识符感知统一预训练编码器-解码器模型

CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation

Salesforce Research Asia(赛富时亚洲研究院);Nanyang Technological University(南洋理工大学)

Yue Wang, Weishi Wang, Shafiq Joty, Steven C. H. Hoi

摘要:

自然语言(NL)(如BERT和GPT)的预训练模型最近被证明可以很好地转移到程序设计语言(PL),并在很大程度上有利于广泛的代码相关任务。尽管它们取得了成功,但大多数当前方法要么依赖于仅编码器(或仅解码器)的预训练,这对于生成(或解码器)而言是次优的。理解)任务或以与NL相同的方式处理代码片段,忽略PL的特殊特性,如令牌类型。我们提出了CodeT 5,这是一个统一的预训练编码器-解码器Transformer模型,可以更好地利用开发人员分配的标识符所传达的代码语义。我们的模型采用了一个统一的框架来无缝地支持代码理解和生成任务,并允许多任务学习。此外,我们提出了一种新的识别器感知的预训练任务,使模型能够区分哪些代码令牌是标识符,并在它们被屏蔽时恢复它们。此外,我们建议利用用户编写的代码注释与双峰双生成任务更好的NL-PL对齐。综合实验表明,CodeT 5在代码缺陷检测和克隆检测等理解任务以及PL-NL、NL-PL和PL-PL等各个方向的生成任务上都明显优于现有方法。进一步的分析表明,我们的模型可以更好地捕捉代码中的语义信息。我们的代码和预训练模型发布在https://github.com/salesforce/CodeT5。

英文摘要:

Pre-trained models for Natural Languages (NL) like BERT and GPT have been recently shown to transfer well to Programming Languages (PL) and largely benefit a broad set of code-related tasks. Despite their success, most current methods either rely on an encoder-only (or decoder-only) pre-training that is suboptimal for generation (resp. understanding) tasks or process the code snippet in the same way as NL, neglecting the special characteristics of PL such as token types. We present CodeT5, a unified pre-trained encoder-decoder Transformer model that better leverages the code semantics conveyed from the developer-assigned identifiers. Our model employs a unified framework to seamlessly support both code understanding and generation tasks and allows for multi-task learning. Besides, we propose a novel identifier-aware pre-training task that enables the model to distinguish which code tokens are identifiers and to recover them when they are masked. Furthermore, we propose to exploit the user-written code comments with a bimodal dual generation task for better NL-PL alignment. Comprehensive experiments show that CodeT5 significantly outperforms prior methods on understanding tasks such as code defect detection and clone detection, and generation tasks across various directions including PL-NL, NL-PL, and PL-PL. Further analysis reveals that our model can better capture semantic information from code. Our code and pre-trained models are released at https: //github.com/salesforce/CodeT5 .

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑