InferCode：通过预测子树实现代码表示的自监督学习
InferCode: Self-Supervised Learning of Code Representations by Predicting Subtrees
摘要：
在源代码上构建深度学习模型已经在许多软件工程应用中取得了成功，如代码搜索、代码注释生成、bug检测、代码迁移等，然而，当前的学习技术存在一个主要缺点，即这些模型大多是在标记为特定下游任务的数据集上训练的，代码表示可能不适合其他任务。虽然一些技术从未标记的代码中产生表示，但当应用于下游任务时，它们远远不能令人满意。尽管某些技术在应用于下游任务时从未标记的代码生成表示，但它们远不能令人满意。本文提出的InferCode通过采用自监督学习机制来建立源代码模型，以克服这一局限性。关键的新颖之处在于通过预测自动识别的子树从AST的上下文中训练代码表示。AST中的子树被InferCode视为训练代码表示的标签，而无需任何人工标记工作或昂贵的图形构造开销，并且训练的表示不再与任何特定的下游任务或代码单元绑定。我们使用基于树的CNN作为大量Java代码的编码器训练了一个InferCode模型实例，并将其应用于下游无监督任务，如代码聚类，代码克隆检测，跨语言代码搜索或在迁移学习方案下重用，以继续训练有监督任务的模型权重，如代码分类和方法名称预测。与以前的代码学习技术应用于相同的下游任务，如Code2Vec，Code2Seq，ASTNN相比，使用我们的预训练的InferCode模型实现了更高的性能结果，对于大多数任务，包括涉及不同程序设计语言的任务，具有显着的裕度。
英文摘要：
Building deep learning models on source code has found many successful software engineering applications, such as code search, code comment generation, bug detection, code migration, and so on. Current learning techniques, however, have a major drawback that these models are mostly trained on datasets labeled for particular downstream tasks, and code representations may not be suitable for other tasks. While some techniques produce representations from unlabeled code, they are far from satisfactory when applied to downstream tasks. Although certain techniques generate representations from unlabeled code when applied to downstream tasks they are far from satisfactory. This paper proposes InferCode to overcome the limitation by adapting the self-supervised learning mechanism to build source code model. The key novelty lies in training code representations by predicting automatically identified subtrees from the context of the ASTs. Subtrees in ASTs are treated with InferCode as the labels for training code representations without any human labeling effort or the overhead of expensive graph construction, and the trained representations are no longer tied to any specific downstream tasks or code units. We trained an InferCode model instance using the Tree-based CNN as the encoder of a large set of Java code and applied it to downstream unsupervised tasks such as code clustering, code clone detection, cross-language code search or reused under a transfer learning scheme to continue training the model weights for supervised tasks such as code classification and method name prediction. Compared to previous code learning techniques applied to the same downstream tasks, such as Code2Vec, Code2Seq, ASTNN, higher performance results are achieved using our pre-trained InferCode model with a significant margin for most tasks including those involving different programming languages.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
