大型语言模型在静态分析中的兴起:通过微基准的首次观察
The Emergence of Large Language Models in Static Analysis: A First Look through Micro-Benchmarks
AI总结:
本文通过PyCG、HeaderGen和TypeEvalPy微基准评估26个LLMs在Python静态分析中的表现,发现其在类型推断上优于传统方法但在调用图分析上存在局限,表明需对LLMs进行专门微调以适配静态分析任务。
摘要:
大型语言模型(LLMs)在软件工程中的应用,特别是在静态分析任务中的应用,代表了该领域的一种范式转变。在本文中,我们研究了当前LLMs在改进Python程序的调用图分析和类型推断方面所能发挥的作用。我们使用PyCG、HeaderGen和TypeEvalPy微基准,评估了26个LLMs,包括OpenAI的GPT系列以及LLaMA等开源模型。我们的研究表明,LLMs在类型推断中展现出有前景的结果,表现出比传统方法更高的准确率,但它们在调用图分析中表现出局限性。这种对比强调了需要对LLMs进行专门的微调,以更好地适应特定的静态分析任务。我们的发现为进一步研究将LLMs集成到静态分析任务中奠定了基础。
英文摘要:
The application of Large Language Models (LLMs) in software engineering, particularly in static analysis tasks, represents a paradigm shift in the field. In this paper, we investigate the role that current LLMs can play in improving callgraph analysis and type inference for Python programs. Using the PyCG, HeaderGen, and TypeEvalPy micro-benchmarks, we evaluate 26 LLMs, including OpenAI's GPT series and open-source models such as LLaMA. Our study reveals that LLMs show promising results in type inference, demonstrating higher accuracy than traditional methods, yet they exhibit limitations in callgraph analysis. This contrast emphasizes the need for specialized fine-tuning of LLMs to better suit specific static analysis tasks. Our findings provide a foundation for further research towards integrating LLMs for static analysis tasks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。