arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2307.12488cs.CRcs.AI
收藏

命名如何影响LLMs在代码分析任务上的表现?

How Does Naming Affect LLMs on Code Analysis Tasks?

College of Information Science and Technology, The Pennsylvania State University(宾夕法尼亚州立大学信息科学与技术学院);Heinz College, Carnegie Mellon University(卡内基梅隆大学海因茨学院)

Zhilong Wang, Lan Zhang, Chen Cao, Nanqing Luo, Xinzhi Luo, Peng Liu

AI总结:

本文研究了程序命名如何影响LLMs在代码分析任务上的表现,通过构建包含无意义或误导性名称的数据集并使用CodeBERT和GPT进行实验,发现命名质量对模型性能有显著影响。

摘要:

大型语言模型(LLMs),如GPT和BERT,是为自然语言处理(NLP)提出的,并作为通用语言模型展现出了可喜的结果。越来越多的行业专业人士和研究人员开始采用LLMs进行程序分析任务。然而,编程语言与自然语言之间一个显著的区别是:程序员可以灵活地为程序中的变量、方法和函数分配任意名称,而自然语言写作者则不能。直觉上,程序中命名的质量会影响LLMs在程序分析任务中的性能。本文研究了命名如何影响LLMs在代码分析任务上的表现。具体而言,我们创建了一组数据集,其中的代码分别包含针对变量、方法和函数的无意义或误导性名称。随后,我们使用训练良好的模型(CodeBERT)在这些数据集上执行代码分析任务。实验结果表明,命名对基于LLMs的代码分析任务性能有显著影响,这表明基于LLMs的代码表示学习严重依赖于代码中定义良好的名称。此外,我们使用GPT对一些特殊的代码分析任务进行了案例研究,提供了进一步的见解。

英文摘要:

The Large Language Models (LLMs), such as GPT and BERT, were proposed for natural language processing (NLP) and have shown promising results as general-purpose language models. An increasing number of industry professionals and researchers are adopting LLMs for program analysis tasks. However, one significant difference between programming languages and natural languages is that a programmer has the flexibility to assign any names to variables, methods, and functions in the program, whereas a natural language writer does not. Intuitively, the quality of naming in a program affects the performance of LLMs in program analysis tasks. This paper investigates how naming affects LLMs on code analysis tasks. Specifically, we create a set of datasets with code containing nonsense or misleading names for variables, methods, and functions, respectively. We then use well-trained models (CodeBERT) to perform code analysis tasks on these datasets. The experimental results show that naming has a significant impact on the performance of code analysis tasks based on LLMs, indicating that code representation learning based on LLMs heavily relies on well-defined names in code. Additionally, we conduct a case study on some special code analysis tasks using GPT, providing further insights.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑