arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2207.08466cs.SEcs.AI
收藏

Transformer了解source code的哪些信息?

What does Transformer learn about source code?

Peking University(北京大学)

Kechi Zhang, Ge Li, Zhi Jin

摘要:

在源代码处理领域,基于transformer的表示模型已经显示出强大的功能,并在许多任务中实现了最先进的(SOTA)性能。尽管Transformer模型处理顺序源代码,但有证据表明,它们也可以捕获结构信息(\eg、语法树、数据流、控制流、\etc)。我们提出了聚合注意力分数,一种方法来调查的结构信息学习的Transformer。我们还提出了聚合注意图,一种新的方法来自动提取程序图从预先训练的模型。我们从多个角度衡量我们的方法。此外,根据我们的实证研究结果,我们使用自动提取的图形,以取代那些巧妙的手动设计的变量误用任务的图形。实验结果表明,自动提取的语义图是有意义和有效的,为我们理解和利用模型中包含的信息提供了一个新的视角。

英文摘要:

In the field of source code processing, the transformer-based representation models have shown great powerfulness and have achieved state-of-the-art (SOTA) performance in many tasks. Although the transformer models process the sequential source code, pieces of evidence show that they may capture the structural information (\eg, in the syntax tree, data flow, control flow, \etc) as well. We propose the aggregated attention score, a method to investigate the structural information learned by the transformer. We also put forward the aggregated attention graph, a new way to extract program graphs from the pre-trained models automatically. We measure our methods from multiple perspectives. Furthermore, based on our empirical findings, we use the automatically extracted graphs to replace those ingenious manual designed graphs in the Variable Misuse task. Experimental results show that the semantic graphs we extracted automatically are greatly meaningful and effective, which provide a new perspective for us to understand and use the information contained in the model.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑