增强用于文本到测试用例生成的大型语言模型
Enhancing Large Language Models for Text-to-Testcase Generation
AI总结:
本文提出一种基于GPT-3.5微调与有效提示设计的文本到测试用例生成方法,在五个开源项目上生成7k测试用例,实现了78.5%语法正确率、67.09%需求对齐率和61.7%代码覆盖率,显著优于其他LLM。
摘要:
背景:测试驱动开发(TDD)是一种广泛采用的软件开发实践,它涉及在编写代码之前根据需求开发测试用例。尽管已经提出了各种自动化测试用例生成方法,但它们并非专门针对TDD定制,在TDD中,需求而非代码作为输入。目标:在本文中,我们引入了一种基于大型语言模型(GPT-3.5)的文本到测试用例生成方法,该模型在我们的精选数据集上进行了微调,并采用了有效的提示设计。方法:我们的方法涉及增强基础GPT-3.5在文本到测试用例生成任务上的能力,该模型在我们的精选数据集上进行了微调,并采用了有效的提示设计。我们在五个大型开源软件项目上评估了我们方法的有效性 结果:我们的方法为开源项目生成了7k个测试用例,实现了78.5%的语法正确率、67.09%的需求对齐率和61.7%的代码覆盖率,显著优于所有其他LLM(基础GPT-3.5、Bloom和CodeT5) 此外,我们的消融研究表明,微调和提示组件对性能有显著提升
英文摘要:
Context: Test-driven development (TDD) is a widely employed software development practice that involves developing test cases based on requirements prior to writing the code. Although various methods for automated test case generation have been proposed, they are not specifically tailored for TDD, where requirements instead of code serve as input. Objective: In this paper, we introduce a text-to-testcase generation approach based on a large language model (GPT-3.5) that is fine-tuned on our curated dataset with an effective prompt design. Method: Our approach involves enhancing the capabilities of basic GPT-3.5 for text-to-testcase generation task that is fine-tuned on our curated dataset with an effective prompting design. We evaluated the effectiveness of our approach using a span of five large-scale open-source software projects. Results: Our approach generated 7k test cases for open source projects, achieving 78.5% syntactic correctness, 67.09% requirement alignment, and 61.7% code coverage, which substantially outperforms all other LLMs (basic GPT-3.5, Bloom, and CodeT5). In addition, our ablation study demonstrates the substantial performance improvement of the fine-tuning and prompting components of the GPT-3.5 model. Conclusions: These findings lead us to conclude that fine-tuning and prompting should be considered in the future when building a language model for the text-to-testcase generation task
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。