COMEX:一个用于生成定制化源代码表示的工具
COMEX: A Tool for Generating Customized Source Code Representations
AI总结:
本文提出COMEX工具,基于tree-sitter从源代码直接生成并组合CFG、DFG、AST等多种代码视图,支持Java和C#及方法级与程序级分析,以弥补LLM忽视代码结构语义的不足,推动ML4SE研究。
摘要:
学习有效的源代码表示对于任何面向软件工程的机器学习(ML4SE)系统都至关重要。受自然语言处理启发,Codex 和 CodeGen 等大型语言模型(LLM)将代码视为通用的文本序列,并在大规模代码数据语料库上进行训练,在多个软件工程(SE)任务上取得了最先进的性能。然而,与自然语言不同,有效的源代码遵循由编程语言的底层语法所支配的严格结构和模式。当前的 LLM 并未利用源代码的这一特性,因为它们将代码视为标记序列,忽视了可以从控制流图(CFG)、数据流图(DFG)、抽象语法树(AST)等代码视图中提取的关键结构和语义属性。不幸的是,为每种编程语言生成和集成代码视图的过程繁琐且耗时。为了克服这一障碍,我们提出了我们的工具 COMEX——一个允许研究人员和开发人员创建和组合多个代码视图的框架,这些代码视图可被机器学习(ML)模型用于各种 SE 任务。我们工具的一些显著特点是:(i)它直接作用于源代码(源代码无需可编译),(ii)它目前支持 Java 和 C#,(iii)它可以通过使用过程内和过程间分析来分析方法级片段和程序级片段,以及(iv)它易于扩展到其他语言,因为它构建在 tree-sitter 之上——一个广泛使用的支持 40 多种语言的增量解析器。我们相信,这个易于使用的代码视图生成和定制工具将推动源代码表示学习方法和 ML4SE 的研究。\n工具:https://pypi.org/project/comex - GitHub:https://github.com/IBM/tree-sitter-codeviews - 演示:https://youtu.be/GER6U87FVbU
英文摘要:
Learning effective representations of source code is critical for any Machine Learning for Software Engineering (ML4SE) system. Inspired by natural language processing, large language models (LLMs) like Codex and CodeGen treat code as generic sequences of text and are trained on huge corpora of code data, achieving state of the art performance on several software engineering (SE) tasks. However, valid source code, unlike natural language, follows a strict structure and pattern governed by the underlying grammar of the programming language. Current LLMs do not exploit this property of the source code as they treat code like a sequence of tokens and overlook key structural and semantic properties of code that can be extracted from code-views like the Control Flow Graph (CFG), Data Flow Graph (DFG), Abstract Syntax Tree (AST), etc. Unfortunately, the process of generating and integrating code-views for every programming language is cumbersome and time consuming. To overcome this barrier, we propose our tool COMEX - a framework that allows researchers and developers to create and combine multiple code-views which can be used by machine learning (ML) models for various SE tasks. Some salient features of our tool are: (i) it works directly on source code (which need not be compilable), (ii) it currently supports Java and C#, (iii) it can analyze both method-level snippets and program-level snippets by using both intra-procedural and inter-procedural analysis, and (iv) it is easily extendable to other languages as it is built on tree-sitter - a widely used incremental parser that supports over 40 languages. We believe this easy-to-use code-view generation and customization tool will give impetus to research in source code representation learning methods and ML4SE. Tool: https://pypi.org/project/comex - GitHub: https://github.com/IBM/tree-sitter-codeviews - Demo: https://youtu.be/GER6U87FVbU
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。