arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2303.06233cs.SE
收藏

预训练程序设计语言模型的模型无关语法信息

Model-Agnostic Syntactical Information for Pre-Trained Programming Language Models

Iman Saberi, Fatemeh H. Fard

摘要:

预训练的程序设计语言模型(PPLM)为许多代码相关的软件工程任务取得了许多最新的最新成果。虽然一些研究使用数据流或提出基于树的模型,利用抽象树(AST),大多数PPLM没有充分利用源代码中丰富的语法信息。尽管如此,输入仍然被认为是一个令牌序列。有两个问题;第一个是由于输入长度和注意力复杂度之间的二次关系导致的计算效率低下。其次,任何语法信息,当需要作为当前PPLM的额外输入时,都需要从头开始对模型进行预训练,浪费了已经用于预训练当前模型的所有计算资源。在这项工作中,我们提出了命名实体识别(NER)适配器,轻量级模块,可以插入到Transformer块学习从AST提取的类型信息。这些适配器可以与当前的PPLM(如CodeBERT、GraphCodeBERT和CodeT 5)一起使用。我们训练NER适配器使用一种新的令牌类型分类目标函数(TTC)。我们将我们提出的工作插入CodeBERT中,构建CodeBERTER,并评估代码细化和代码摘要两项任务的性能。与完全微调方法相比,CodeBERTER将代码细化的准确性从16.4提高到17.8,同时使用20%的训练参数预算,并且与完全微调方法相比,代码摘要的BLEU得分从14.75提高到15.90,同时减少77%的训练参数。

英文摘要:

Pre-trained Programming Language Models (PPLMs) achieved many recent states of the art results for many code-related software engineering tasks. Though some studies use data flow or propose tree-based models that utilize Abstract Syntax Tree (AST), most PPLMs do not fully utilize the rich syntactical information in source code. Still, the input is considered a sequence of tokens. There are two issues; the first is computational inefficiency due to the quadratic relationship between input length and attention complexity. Second, any syntactical information, when needed as an extra input to the current PPLMs, requires the model to be pre-trained from scratch, wasting all the computational resources already used for pre-training the current models. In this work, we propose Named Entity Recognition (NER) adapters, lightweight modules that can be inserted into Transformer blocks to learn type information extracted from the AST. These adapters can be used with current PPLMs such as CodeBERT, GraphCodeBERT, and CodeT5. We train the NER adapters using a novel Token Type Classification objective function (TTC). We insert our proposed work in CodeBERT, building CodeBERTER, and evaluate the performance on two tasks of code refinement and code summarization. CodeBERTER improves the accuracy of code refinement from 16.4 to 17.8 while using 20% of training parameter budget compared to the fully fine-tuning approach, and the BLEU score of code summarization from 14.75 to 15.90 while reducing 77% of training parameters compared to the fully fine-tuning approach.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑