基于BiLSTM语句嵌入的语句级语言模型改进代码完成语言建模
Improve Language Modelling for Code Completion through Statement Level Language Model based on Statement Embedding Generated by BiLSTM
摘要:
诸如RNN、LSTM或其他变体的语言模型已被广泛用作自然语言处理中的生成模型。近年来,将源代码看作自然语言,将源代码解析成一个符号序列,并使用LSTM等语言模型对该序列进行训练,得到一个生成模型来解决代码补全问题。然而,对于包含数百个语句的源代码,传统的LSTM模型或基于注意力的LSTM模型无法捕获源代码的长期依赖性。在本文中,我们提出了一种新的语句级语言模型(SLM),它使用BiLSTM为每个语句生成嵌入。SLM中采用标准的LSTM,以在上下文中对每个语句的嵌入进行筛选和累积,以帮助预测下一个代码。模型中还采用了语句级注意力机制。所提出的模型SLM是针对令牌级代码完成。在项目内和跨项目数据集上的实验表明,新提出的带有注意力机制(SLM)的语句级语言模型在令牌级代码完成任务中的性能优于其他现有模型.
英文摘要:
Language models such as RNN, LSTM or other variants have been widely used as generative models in natural language processing. In last few years, taking source code as natural languages, parsing source code into a token sequence and using a language model such as LSTM to train that sequence are state-of-art methods to get a generative model for solving the problem of code completion. However, for source code with hundreds of statements, traditional LSTM model or attention-based LSTM model failed to capture the long term dependency of source code. In this paper, we propose a novel statement-level language model (SLM) which uses BiLSTM to generate the embedding for each statement. The standard LSTM is adopted in SLM to iterate and accumulate the embedding of each statement in context to help predict next code. The statement level attention mechanism is also adopted in the model. The proposed model SLM is aimed at token level code completion. The experiments on inner-project and cross-project data sets indicate that the newly proposed statement-level language model with attention mechanism (SLM) outperforms all other state-of-art models in token level code completion task.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。