arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2304.11686cs.SE
收藏

细节是关键:通过差分提示解锁ChatGPT以寻找触发失效的测试用例

Nuances are the Key: Unlocking ChatGPT to Find Failure-Inducing Tests with Differential Prompting

Tsz-On Li, Wenxi Zong, Yibo Wang, Haoye Tian, Ying Wang, Shing-Chi Cheung, Jeff Kramer

AI总结:

针对ChatGPT难以识别细微代码差异导致寻找触发失效测试用例概率低的问题,提出协同结合ChatGPT和差分测试的新方法,通过推断预期行为合成程序来凸显代码差异,将成功率提升至77.8%。

摘要:

自动检测软件失效是一项重要任务,也是一个长期存在的挑战。它需要找到触发失效的测试用例,其测试输入能触发软件的故障,并构建自动化的预言机来检测软件的不正确行为。大型语言模型(LLMs)的最新进展促使我们研究ChatGPT(一种最先进的LLM)能在多大程度上解决这一挑战。遗憾的是,我们的研究表明,ChatGPT为缺陷程序找到正确触发失效的测试用例的概率很低(28.8%)。一个可能的原因是,寻找触发失效的测试用例需要分析缺陷程序与其正确版本之间细微的代码差异。当这两个版本具有相似的语法时,ChatGPT在识别细微代码差异方面表现较弱。我们的洞察是,当引导ChatGPT专注于细微代码差异时,其性能可以得到大幅提升。我们有一个有趣的观察:ChatGPT能有效推断缺陷程序的预期行为。该预期行为可用于合成程序,以使缺陷程序与其正确版本(即合成程序)之间的细微代码差异变得明确。基于这一观察,我们提出了一种新方法,该方法协同结合ChatGPT和差分测试来寻找触发失效的测试用例。我们在Quixbugs(一个缺陷程序基准)上评估了我们的方法,并将其与最先进的基线进行比较,包括直接使用ChatGPT和Pynguin。实验结果表明,我们的方法找到正确触发失效的测试用例的概率要高得多(77.8%),是最优基线的2.7倍。

英文摘要:

Automatically detecting software failures is an important task and a longstanding challenge. It requires finding failure-inducing test cases whose test input can trigger the software's fault, and constructing an automated oracle to detect the software's incorrect behaviors. Recent advancement of large language models (LLMs) motivates us to study how far this challenge can be addressed by ChatGPT, a state-of-the-art LLM. Unfortunately, our study shows that ChatGPT has a low probability (28.8%) of finding correct failure-inducing test cases for buggy programs. A possible reason is that finding failure-inducing test cases requires analyzing the subtle code differences between a buggy program and its correct version. When these two versions have similar syntax, ChatGPT is weak at recognizing subtle code differences. Our insight is that ChatGPT's performance can be substantially enhanced when ChatGPT is guided to focus on the subtle code difference. We have an interesting observation that ChatGPT is effective in inferring the intended behaviors of a buggy program. The intended behavior can be leveraged to synthesize programs, in order to make the subtle code difference between a buggy program and its correct version (i.e., the synthesized program) explicit. Driven by this observation, we propose a novel approach that synergistically combines ChatGPT and differential testing to find failure-inducing test cases. We evaluate our approach on Quixbugs (a benchmark of buggy programs), and compare it with state-of-the-art baselines, including direct use of ChatGPT and Pynguin. The experimental result shows that our approach has a much higher probability (77.8%) of finding correct failure-inducing test cases, 2.7X as the best baseline.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑