使用文本到文本转换Transformer支持代码相关任务的研究
Studying the Usage of Text-To-Text Transfer Transformer to Support Code-Related Tasks
摘要：
深度学习（DL）技术在软件工程社区中受到越来越多的关注。它们已被用于支持几个与代码相关的任务，例如自动修复错误和生成代码注释。自然语言处理（NLP）领域的最新研究表明，文本到文本传输Transformer（T5）架构可以为各种NLP任务实现最先进的性能。T5背后的基本思想是首先使用自监督任务（例如：在句子中填充掩码词）在大型通用数据集上预训练模型。一旦模型经过预训练，它就会在较小和专业的数据集上进行微调，每个数据集都与特定任务相关（例如：语言翻译，句子分类）。在本文中，我们实证研究了T5模型在经过预训练和微调以支持代码相关任务时的表现。我们在由自然语言英文文本和源代码组成的数据集上预训练T5模型。然后，我们通过重用之前四个使用DL技术的作品中使用的数据集来微调这样的模型：（i）修复错误，（ii）注入代码突变体，（iii）生成断言语句，以及（iv）生成代码注释。我们比较了这个单一模型的性能与四篇原始论文中报告的结果，这些论文为这四项任务提出了基于DL的解决方案。我们证明了我们的T5模型，利用额外的数据进行自我监督的预训练阶段，可以在四个基线上实现性能改进。
英文摘要：
Deep learning (DL) techniques are gaining more and more attention in the software engineering community. They have been used to support several code-related tasks, such as automatic bug fixing and code comments generation. Recent studies in the Natural Language Processing (NLP) field have shown that the Text-To-Text Transfer Transformer (T5) architecture can achieve state-of-the-art performance for a variety of NLP tasks. The basic idea behind T5 is to first pre-train a model on a large and generic dataset using a self-supervised task ( e.g: filling masked words in sentences). Once the model is pre-trained, it is fine-tuned on smaller and specialized datasets, each one related to a specific task ( e.g: language translation, sentence classification). In this paper, we empirically investigate how the T5 model performs when pre-trained and fine-tuned to support code-related tasks. We pre-train a T5 model on a dataset composed of natural language English text and source code. Then, we fine-tune such a model by reusing datasets used in four previous works that used DL techniques to: (i) fix bugs, (ii) inject code mutants, (iii) generate assert statements, and (iv) generate code comments. We compared the performance of this single model with the results reported in the four original papers proposing DL-based solutions for those four tasks. We show that our T5 model, exploiting additional data for the self-supervised pre-training phase, can achieve performance improvements over the four baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
