arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.17019cs.SE
收藏

使用大型语言模型生成可执行的蜕变关系

Towards Generating Executable Metamorphic Relations Using Large Language Models

Seung Yeob Shin, Fabrizio Pastore, Domenico Bianculli, Alexandra Baicoianu

AI总结:

本文提出一种基于大型语言模型和少样本提示策略,从需求自动推导可执行蜕变关系的方法,实验证明其生成的蜕变关系具有高准确性和实用价值。

摘要:

蜕变测试(MT)已被证明是自动化测试和解决预言机问题的成功方案。然而,它需要手动推导蜕变关系(MRs)并将其转换为可执行形式;这些步骤非常耗时,可能会阻碍MT的采用。在本文中,我们提出了一种使用大型语言模型(LLMs)从需求中自动推导可执行蜕变关系(EMRs)的方法。我们的方法并非仅仅要求LLM生成EMRs,而是依赖一种少样本提示策略,通过提供需求和API规范,指导LLM像软件工程师一样执行MT过程中的活动。为了评估我们方法的可行性,我们与行业软件和服务全球领导者Siemens Industry Software合作,开展了一项基于问卷的调查,重点关注其四款软件应用。此外,我们还评估了针对Web应用生成的EMRs的准确性。研究结果非常有前景,证明了我们的方法能够生成易于理解且与测试目的相关的MRs和EMRs。

英文摘要:

Metamorphic testing (MT) has proven to be a successful solution to automating testing and addressing the oracle problem. However, it entails manually deriving metamorphic relations (MRs) and converting them into an executable form; these steps are time-consuming and may prevent the adoption of MT. In this paper, we propose an approach for automatically deriving executable MRs (EMRs) from requirements using large language models (LLMs). Instead of merely asking the LLM to produce EMRs, our approach relies on a few-shot prompting strategy to instruct the LLM to perform activities in the MT process, by providing requirements and API specifications, as one would do with software engineers. To assess the feasibility of our approach, we conducted a questionnaire-based survey in collaboration with Siemens Industry Software, a worldwide leader in providing industry software and services, focusing on four of their software applications. Additionally, we evaluated the accuracy of the generated EMRs for a Web application. The outcomes of our study are highly promising, as they demonstrate the capability of our approach to generate MRs and EMRs that are both comprehensible and pertinent for testing purposes.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑