如何利用代码预训练模型获得更好的嵌入？一项实证研究
How to get better embeddings with code pre-trained models? An empirical study
AI总结：
本文实证研究了五种代码预训练模型在四个软件工程分类任务上的嵌入质量，发现特殊标记嵌入不充分、预训练方式组合数据效果差，而聚合所有代码标记的向量表示可使仅解码器模型获得更优嵌入。
摘要：
预训练语言模型已在自然语言处理（NLP）领域展现出强大的能力。近年来，借鉴NLP领域经验的代码预训练模型（PTM）也在许多软件工程（SE）下游任务中取得了最先进的成果。这些代码PTM在预训练期间考虑了编程语言与自然语言之间的差异，并对预训练任务和输入数据进行了调整。然而，SE社区的研究人员在使用这些代码PTM为SE下游分类任务生成嵌入时，仍继承了NLP领域的习惯，例如通过特殊标记为代码片段生成语义嵌入，以及以与PTM预训练相同的方式输入代码和文本信息。在本文中，我们针对上述两个方面，对五种不同的PTM（即CodeBERT、CodeT5、PLBART、CodeGPT和CodeGen）以及三种不同的架构（即仅编码器、仅解码器和编码器-解码器）在四个SE下游分类任务（即代码漏洞检测、代码克隆检测、即时缺陷预测和函数文档字符串不匹配检测）上进行了实证研究。我们的实验结果表明：（1）无论使用何种代码PTM架构，通过特殊标记获得的嵌入都不能充分聚合整个代码片段的语义信息；（2）通过以与PTM预训练相同的方式组合代码数据和文本数据而获得的代码嵌入质量较差，无法保证更丰富的语义信息；（3）使用聚合所有代码标记的向量表示的方法，仅解码器PTM可以获得与仅编码器和编码器-解码器PTM获得的语义同样丰富甚至质量更好的代码嵌入。
英文摘要：
Pre-trained language models have demonstrated powerful capabilities in the field of natural language processing (NLP). Recently, code pre-trained model (PTM), which draw from the experiences of the NLP field, have also achieved state-of-the-art results in many software engineering (SE) downstream tasks. These code PTMs take into account the differences between programming languages and natural languages during pre-training and make adjustments to pre-training tasks and input data. However, researchers in the SE community still inherit habits from the NLP field when using these code PTMs to generate embeddings for SE downstream classification tasks, such as generating semantic embeddings for code snippets through special tokens and inputting code and text information in the same way as pre-training the PTMs. In this paper, we empirically study five different PTMs (i.e. CodeBERT, CodeT5, PLBART, CodeGPT and CodeGen) with three different architectures (i.e. encoder-only, decoder-only and encoder-decoder) on four SE downstream classification tasks (i.e. code vulnerability detection, code clone detection, just-in-time defect prediction and function docstring mismatch detection) with respect to the two aforementioned aspects. Our experimental results indicate that (1) regardless of the architecture of the code PTMs used, embeddings obtained through special tokens do not sufficiently aggregate the semantic information of the entire code snippet; (2) the quality of code embeddings obtained by combing code data and text data in the same way as pre-training the PTMs is poor and cannot guarantee richer semantic information; (3) using the method that aggregates the vector representations of all code tokens, the decoder-only PTMs can obtain code embeddings with semantics as rich as or even better quality than those obtained from the encoder-only and encoder-decoder PTMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
