用层次语言模型提高代码完成训练与测试对数据不一致的鲁棒性
Improving the Robustness to Data Inconsistency between Training and Testing for Code Completion by Hierarchical Language Model
摘要：
在软件工程领域，将语言模型应用于源代码的标记序列是构建代码推荐系统的一种最新方法。源代码的语法树具有层次结构。忽略树结构的特性会降低模型的性能。当前的LSTM模型处理顺序数据。如果噪声不可见的数据分布在测试集中的任何地方，LSTM模型的性能将急剧下降。由于代码有自由的命名约定，在一个项目上训练的模型在另一个项目上遇到许多未知词是很常见的。如果我们像自然语言处理中的解决方案一样，将许多未见过的单词设置为UNK，则UNK的数量将远远大于最频繁出现的单词的总和。在极端情况下，仅在任何地方预测UNK可以实现非常高的预测精度。因此，当遇到噪声不可见数据时，这种解决方案不能反映模型的真实性能。在本文中，我们只将少数罕见词标记为UNK，并展示了模型在项目内和跨项目评估下的预测性能。提出了一种新的层次语言模型（HLM）来提高LSTM模型的鲁棒性，以获得处理训练和测试数据分布不一致的能力。新提出的HLM考虑到代码树的层次结构来预测代码。HLM使用BiLSTM根据层次结构生成子树的嵌入，并收集上下文中的子树嵌入以预测下一个代码。在项目内和跨项目数据集上的实验表明，新提出的分层语言模型（HLM）在处理训练和测试数据不一致性方面优于现有的LSTM模型，预测精度平均提高11.2%.
英文摘要：
In the field of software engineering, applying language models to the token sequence of source code is the state-of-art approach to build a code recommendation system. The syntax tree of source code has hierarchical structures. Ignoring the characteristics of tree structures decreases the model performance. Current LSTM model handles sequential data. The performance of LSTM model will decrease sharply if the noise unseen data is distributed everywhere in the test suite. As code has free naming conventions, it is common for a model trained on one project to encounter many unknown words on another project. If we set many unseen words as UNK just like the solution in natural language processing, the number of UNK will be much greater than the sum of the most frequently appeared words. In an extreme case, just predicting UNK at everywhere may achieve very high prediction accuracy. Thus, such solution cannot reflect the true performance of a model when encountering noise unseen data. In this paper, we only mark a small number of rare words as UNK and show the prediction performance of models under in-project and cross-project evaluation. We propose a novel Hierarchical Language Model (HLM) to improve the robustness of LSTM model to gain the capacity about dealing with the inconsistency of data distribution between training and testing. The newly proposed HLM takes the hierarchical structure of code tree into consideration to predict code. HLM uses BiLSTM to generate embedding for sub-trees according to hierarchies and collects the embedding of sub-trees in context to predict next code. The experiments on inner-project and cross-project data sets indicate that the newly proposed Hierarchical Language Model (HLM) performs better than the state-of-art LSTM model in dealing with the data inconsistency between training and testing and achieves averagely 11.2\% improvement in prediction accuracy.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
