通过神经语言模型探索软件自然性
Exploring Software Naturalness through Neural Language Models
摘要：
软件自然性假设认为，程序设计语言可以通过自然语言处理中使用的相同技术来理解。我们通过使用预先训练的基于transformer的语言模型来执行代码分析任务来探索这一假设。目前的代码分析方法在很大程度上依赖于从抽象树（AST）的功能，而我们的基于transformer的语言模型的原始源代码的工作。这项工作是第一次调查是否这样的语言模型可以自动发现AST功能。为了实现这一点，我们引入了一个序列标记任务，直接探测的语言模型理解AST。我们的研究结果表明，Transformer为基础的语言模型实现高精度的AST标注任务。此外，我们评估我们的模型上的软件漏洞识别任务。重要的是，我们表明，我们的方法获得的漏洞识别结果相比，严重依赖编译器的特征提取基于图的方法。
英文摘要：
The Software Naturalness hypothesis argues that programming languages can be understood through the same techniques used in natural language processing. We explore this hypothesis through the use of a pre-trained transformer-based language model to perform code analysis tasks. Present approaches to code analysis depend heavily on features derived from the Abstract Syntax Tree (AST) while our transformer-based language models work on raw source code. This work is the first to investigate whether such language models can discover AST features automatically. To achieve this, we introduce a sequence labeling task that directly probes the language models understanding of AST. Our results show that transformer based language models achieve high accuracy in the AST tagging task. Furthermore, we evaluate our model on a software vulnerability identification task. Importantly, we show that our approach obtains vulnerability identification results comparable to graph based approaches that rely heavily on compilers for feature extraction.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
