ContraBERT：通过对比学习增强代码预训练模型
ContraBERT: Enhancing Code Pre-trained Models via Contrastive Learning
摘要：
CodeBERT、GraphCodeBERT等大规模预训练模型已经引起了学术界和工业界的广泛关注。由于其在代码表示方面的优越能力，它们已被进一步应用于克隆检测、代码搜索和代码翻译等多个下游任务。然而，还观察到这些最先进的预训练模型容易受到对抗攻击的影响。这些预先训练好的模型的性能会因为简单的扰动而显著下降，比如重命名变量名。这种弱点可能会被他们的下游模型继承，从而以前所未有的规模放大。为此，我们提出了一种方法，即ContraBERT，旨在通过对比学习提高预训练模型的鲁棒性。具体而言，我们设计了九种简单和复杂的数据增强操作符上的程序设计语言（PL）和自然语言（NL）的数据构建不同的变种。此外，我们继续训练现有的预训练模型的掩蔽语言建模（MLM）和对比预训练任务的原始样本与他们的增强变体，以提高模型的鲁棒性。大量的实验表明，ContraBERT可以有效地提高现有预训练模型的鲁棒性。进一步的研究还证实，这些鲁棒性增强的模型提供了改进相比，原始模型在四个流行的下游任务。
英文摘要：
Large-scale pre-trained models such as CodeBERT, GraphCodeBERT have earned widespread attention from both academia and industry. Attributed to the superior ability in code representation, they have been further applied in multiple downstream tasks such as clone detection, code search and code translation. However, it is also observed that these state-of-the-art pre-trained models are susceptible to adversarial attacks. The performance of these pre-trained models drops significantly with simple perturbations such as renaming variable names. This weakness may be inherited by their downstream models and thereby amplified at an unprecedented scale. To this end, we propose an approach namely ContraBERT that aims to improve the robustness of pre-trained models via contrastive learning. Specifically, we design nine kinds of simple and complex data augmentation operators on the programming language (PL) and natural language (NL) data to construct different variants. Furthermore, we continue to train the existing pre-trained models by masked language modeling (MLM) and contrastive pre-training task on the original samples with their augmented variants to enhance the robustness of the model. The extensive experiments demonstrate that ContraBERT can effectively improve the robustness of the existing pre-trained models. Further study also confirms that these robustness-enhanced models provide improvements as compared to original models over four popular downstream tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
