通过代码转换深入审视基于 Transformer 的代码智能:挑战与机遇
A Closer Look into Transformer-Based Code Intelligence Through Code Transformation: Challenges and Opportunities
AI总结:
本文针对 Java 与 Python 设计多类语义保持代码转换,系统评估 Transformer 在代码补全、摘要和搜索任务中的鲁棒性,并揭示 AST、标识符扰动及位置编码的影响与改进方向。
摘要:
基于 Transformer 的模型已在代码注释生成和代码补全等许多智能编码任务中展现出最先进的性能。以往研究表明,深度学习模型对输入变化很敏感,但很少有研究系统地考察 Transformer 在受扰动输入代码下的鲁棒性。在本工作中,我们通过实证研究保持语义不变的代码转换对 Transformer 性能的影响。具体而言,我们分别针对两种流行编程语言 Java 和 Python 实现了 24 种和 27 种代码转换策略。为便于分析,这些策略被归为五类:块转换、插入/删除转换、语法语句转换、语法 token 转换以及标识符转换。在代码补全、代码摘要和代码搜索这三个流行代码智能任务上的实验表明,插入/删除转换和标识符转换对 Transformer 性能的影响最大。我们的结果还表明,在大多数代码转换下,基于抽象语法树(AST)的 Transformer 比仅基于代码序列的模型表现出更强的鲁棒性。此外,位置编码的设计会影响 Transformer 在代码转换下的鲁棒性。基于这些发现,我们提炼出关于基于 Transformer 的代码智能所面临挑战与机遇的一些洞见。
英文摘要:
Transformer-based models have demonstrated state-of-the-art performance in many intelligent coding tasks such as code comment generation and code completion. Previous studies show that deep learning models are sensitive to the input variations, but few studies have systematically studied the robustness of Transformer under perturbed input code. In this work, we empirically study the effect of semantic-preserving code transformation on the performance of Transformer. Specifically, 24 and 27 code transformation strategies are implemented for two popular programming languages, Java and Python, respectively. For facilitating analysis, the strategies are grouped into five categories: block transformation, insertion/deletion transformation, grammatical statement transformation, grammatical token transformation, and identifier transformation. Experiments on three popular code intelligence tasks, including code completion, code summarization and code search, demonstrate insertion/deletion transformation and identifier transformation show the greatest impact on the performance of Transformer. Our results also suggest that Transformer based on abstract syntax trees (ASTs) shows more robust performance than the model based on only code sequence under most code transformations. Besides, the design of positional encoding can impact the robustness of Transformer under code transformation. Based on our findings, we distill some insights about the challenges and opportunities for Transformer-based code intelligence.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。