神经程序模型在语义保持程序变换中的泛化性
On the Generalizability of Neural Program Models with respect to Semantic-Preserving Program Transformations
摘要：
随着用于训练深度神经网络模型的公共可用源代码存储库的流行，神经程序模型可以在源代码分析任务中做得很好，例如预测给定程序中的方法名称，这是传统程序分析技术无法轻松完成的。虽然这种神经程序模型已经在各种现有的数据集上进行了测试，但它们在多大程度上推广到不可预见的源代码在很大程度上是未知的。由于在所有不可预见的程序上测试神经程序模型是非常具有挑战性的，因此在本文中，我们建议评估神经程序模型在语义保持转换方面的可推广性：可推广的神经程序模型应该在具有相同语义但具有不同词汇外观和语法结构的程序上同样表现良好。我们比较了各种神经程序模型在自动语义保持转换之前和之后的程序上的方法名称预测任务的结果。我们使用三个不同大小的Java数据集和三个最先进的神经网络代码模型，即code2vec，code2seq和GGNN，来构建九个这样的神经程序模型进行评估。我们的研究结果表明，即使对程序进行小的语义保留更改，这些神经程序模型通常也无法概括其性能。我们的研究结果还表明，基于程序中的数据和控制依赖关系的神经程序模型比仅基于抽象语法树的神经程序模型更好地泛化。从积极的方面来看，我们观察到，随着训练数据集的大小增加和多样化，神经程序模型产生的正确预测的泛化能力也可以提高。我们关于神经程序模型的可推广性的结果提供了衡量其局限性的见解，并为其改进提供了垫脚石。
英文摘要：
With the prevalence of publicly available source code repositories to train deep neural network models, neural program models can do well in source code analysis tasks such as predicting method names in given programs that cannot be easily done by traditional program analysis techniques. Although such neural program models have been tested on various existing datasets, the extent to which they generalize to unforeseen source code is largely unknown. Since it is very challenging to test neural program models on all unforeseen programs, in this paper, we propose to evaluate the generalizability of neural program models with respect to semantic-preserving transformations: a generalizable neural program model should perform equally well on programs that are of the same semantics but of different lexical appearances and syntactical structures. We compare the results of various neural program models for the method name prediction task on programs before and after automated semantic-preserving transformations. We use three Java datasets of different sizes and three state-of-the-art neural network models for code, namely code2vec, code2seq, and GGNN, to build nine such neural program models for evaluation. Our results show that even with small semantically preserving changes to the programs, these neural program models often fail to generalize their performance. Our results also suggest that neural program models based on data and control dependencies in programs generalize better than neural program models based only on abstract syntax trees. On the positive side, we observe that as the size of the training dataset grows and diversifies the generalizability of correct predictions produced by the neural program models can be improved too. Our results on the generalizability of neural program models provide insights to measure their limitations and provide a stepping stone for their improvement.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
