arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.10375cs.SE
收藏

Mokav:利用大语言模型的执行驱动差分测试

Mokav: Execution-driven Differential Testing with LLMs

Khashayar Etemadi, Bardia Mohammadi, Zhendong Su, Martin Monperrus

AI总结:

针对程序功能差异检测问题,提出执行驱动工具Mokav,通过迭代结合执行反馈引导LLM生成差异暴露测试用例,在基准测试上性能显著优于现有方法。

摘要:

在自动化程序修复、变异测试、代码重构等各类软件工程任务中,检测程序间的功能差异至关重要。检测两个程序间功能差异的问题可归结为搜索差异暴露测试用例(DET):即能使被测程序产生不同输出的测试输入。本文提出Mokav,这是一款新颖的执行驱动工具,利用大语言模型(LLM)生成DET。Mokav接收两个版本的程序(P和Q)以及一个示例测试输入,成功时会生成有效的DET——即能让P和Q产生可证实不同输出的测试输入。Mokav通过专用提示词迭代地向LLM发起请求以生成新的测试输入,在每一轮迭代中,它都会提供基于先前生成测试的执行反馈,直到LLM生成DET为止。我们在从Codeforces竞赛平台收集的1535对Python程序,以及QuixBugs数据集的32对程序上评估了Mokav。实验表明,Mokav的性能大幅优于现有最优方法Pynguin和Differential Prompting:在我们的基准测试中,Mokav能为81.7%(1255/1535)的程序对生成DET,而Pynguin仅为4.9%,Differential Prompting为37.3%。我们证实,该系统的迭代机制和执行驱动反馈组件是其高效性的重要来源。

英文摘要:

It is essential to detect functional differences between programs in various software engineering tasks, such as automated program repair, mutation testing, and code refactoring. The problem of detecting functional differences between two programs can be reduced to searching for a difference exposing test (DET): a test input that results in different outputs on the subject programs. In this paper, we propose Mokav, a novel execution-driven tool that leverages LLMs to generate DETs. Mokav takes two versions of a program (P and Q) and an example test input. When successful, Mokav generates a valid DET, a test input that leads to provably different outputs on P and Q. Mokav iteratively prompts an LLM with a specialized prompt to generate new test inputs. At each iteration, Mokav provides execution-based feedback from previously generated tests until the LLM produces a DET. We evaluate Mokav on 1535 pairs of Python programs collected from the Codeforces competition platform and 32 pairs of programs from the QuixBugs dataset. Our experiments show that Mokav outperforms the state-of-the-art, Pynguin and Differential Prompting, by a large margin. Mokav can generate DETs for 81.7% (1,255/1535) of the program pairs in our benchmark (versus 4.9% for Pynguin and 37.3% for Differential Prompting). We demonstrate that the iterative and execution-driven feedback components of the system contribute to its high effectiveness.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑