他们捕捉什么？--Source Code预训练语言模型的结构分析
What Do They Capture? -- A Structural Analysis of Pre-Trained Language Models for Source Code
摘要：
最近，已经提出了许多用于源代码的预训练的语言模型来对代码的上下文进行建模，并用作下游代码智能任务（诸如代码完成、代码搜索和代码摘要）的基础。这些模型利用了掩码预训练和Transformer，并取得了可喜的成果。然而，目前在现有预训练代码模型的可解释性方面仍然进展甚微。目前尚不清楚这些模型为什么有效，以及它们可以捕获哪些特征相关性。在本文中，我们进行了彻底的结构分析，旨在为源代码提供预训练的语言模型的解释（例如，CodeBERT和GraphCodeBERT）从三个不同的角度：（1）注意力分析，（2）对词嵌入的探索，（3）语法树归纳。通过综合分析，本文揭示了几个有启发意义的发现：（1）注意与代码的语法结构密切相关。(2)代码的预训练语言模型可以在每个Transformer层的中间表示中保留代码的语法结构。(3)预训练的代码模型具有归纳代码语法树的能力。这些发现表明，将代码的语法结构结合到预训练过程中以获得更好的代码表示可能会有所帮助。
英文摘要：
Recently, many pre-trained language models for source code have been proposed to model the context of code and serve as a basis for downstream code intelligence tasks such as code completion, code search, and code summarization. These models leverage masked pre-training and Transformer and have achieved promising results. However, currently there is still little progress regarding interpretability of existing pre-trained code models. It is not clear why these models work and what feature correlations they can capture. In this paper, we conduct a thorough structural analysis aiming to provide an interpretation of pre-trained language models for source code (e.g., CodeBERT, and GraphCodeBERT) from three distinctive perspectives: (1) attention analysis, (2) probing on the word embedding, and (3) syntax tree induction. Through comprehensive analysis, this paper reveals several insightful findings that may inspire future studies: (1) Attention aligns strongly with the syntax structure of code. (2) Pre-training language models of code can preserve the syntax structure of code in the intermediate representations of each Transformer layer. (3) The pre-trained models of code have the ability of inducing syntax trees of code. Theses findings suggest that it may be helpful to incorporate the syntax structure of code into the process of pre-training for better code representations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
