arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.05940cs.SEcs.AI
收藏

基于变异的语义一致性测试：评估大语言模型的代码理解能力

Mutation-based Consistency Testing for Evaluating the Code Understanding Capability of LLMs

University of Sheffield（谢菲尔德大学）

Ziyu Li, Donghwan Shin

备注：
This is an author-preprint. The published version will be included in the proceedings of CAIN 2024 (co-located with ICSE 2024)
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出基于变异的语义一致性测试方法MCT，通过向HumanEval-X数据集引入运算符替换和语句删除等代码变异来评估GPT-3.5和GPT-4的代码理解能力，发现不同模型在不同变异类型和编程语言上表现差异显著。

摘要：

大语言模型（LLMs）在处理自然语言和编程语言方面展现出卓越的能力，这推动了软件工程领域的多种应用，如需求工程、代码生成和软件测试。然而，现有的代码生成基准并不一定能评估LLMs的代码理解性能，尤其是对于代码与其自然语言描述之间可能出现的细微不一致。本文提出了一种新方法，通过向现有代码生成数据集引入代码变异，系统地评估LLMs的代码理解性能，特别关注代码与其描述之间的细微差异。代码变异是指改变原始代码语义的小改动，从而造成与自然语言描述的不匹配。我们应用不同类型的代码变异，如运算符替换和语句删除，来生成不一致的代码-描述对。然后，我们使用这些配对来测试LLMs正确检测不一致的能力。我们提出了一种新的LLM测试方法，称为基于变异的语义一致性测试（Mutation-based Consistency Testing，MCT），并使用最先进的代码生成基准HumanEval-X进行了案例研究，该基准包含六种编程语言（Python、C++、Java、Go、JavaScript和Rust），测试对象为两个流行的LLM：GPT-3.5和GPT-4。我们比较了LLMs在不同类型的代码变异和编程语言上的表现，并分析了结果。我们发现，LLMs在代码理解性能上表现出显著差异，并且根据变异类型和语言的不同，它们具有不同的优势和劣势。

英文摘要：

Large Language Models (LLMs) have shown remarkable capabilities in processing both natural and programming languages, which have enabled various applications in software engineering, such as requirement engineering, code generation, and software testing. However, existing code generation benchmarks do not necessarily assess the code understanding performance of LLMs, especially for the subtle inconsistencies that may arise between code and its semantics described in natural language. In this paper, we propose a novel method to systematically assess the code understanding performance of LLMs, particularly focusing on subtle differences between code and its descriptions, by introducing code mutations to existing code generation datasets. Code mutations are small changes that alter the semantics of the original code, creating a mismatch with the natural language description. We apply different types of code mutations, such as operator replacement and statement deletion, to generate inconsistent code-description pairs. We then use these pairs to test the ability of LLMs to correctly detect the inconsistencies. We propose a new LLM testing method, called Mutation-based Consistency Testing (MCT), and conduct a case study on the two popular LLMs, GPT-3.5 and GPT-4, using the state-of-the-art code generation benchmark, HumanEval-X, which consists of six programming languages (Python, C++, Java, Go, JavaScript, and Rust). We compare the performance of the LLMs across different types of code mutations and programming languages and analyze the results. We find that the LLMs show significant variation in their code understanding performance and that they have different strengths and weaknesses depending on the mutation type and language.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑