PyTester:用于文本到测试用例生成的深度强化学习
PyTester: Deep Reinforcement Learning for Text-to-Testcase Generation
AI总结:
PyTester利用深度强化学习,从自然语言需求直接生成测试用例,在APPS基准上以小型语言模型超越GPT3.5等大型模型,提升资源效率。
摘要:
测试驱动开发(TDD)是一种广泛使用的软件开发实践,要求在编写实际代码之前根据需求编写测试用例。虽然编写测试用例是TDD的核心,但它耗时、昂贵,且常常被开发者回避。为了解决这些与TDD相关的问题,近年来研究了自动化测试用例生成方法。这些方法以源代码作为输入,而非需求。因此,现有工作并未完全支持真正的TDD,因为生成测试用例需要实际代码。此外,当前基于深度学习的测试用例生成方法使用单一学习目标进行训练,即生成与真实测试用例完全匹配的测试用例。然而,这种方法可能限制模型生成不同但正确测试用例的能力。在本文中,我们介绍了PyTester,一种文本到测试用例生成方法,可以自动生成语法正确、可执行、完整且有效的测试用例,同时与给定的自然语言需求保持一致。我们在公开的APPS基准数据集上评估了PyTester,结果表明,我们的深度强化学习方法使PyTester(一个小型语言模型)能够超越更大的语言模型,如GPT3.5、StarCoder和InCoder。我们的发现表明,未来的研究可以考虑通过将软件工程领域知识整合到强化学习架构的设计中,来改进小型语言模型而非大型语言模型,以实现更好的资源效率。
英文摘要:
Test-driven development (TDD) is a widely-employed software development practice that mandates writing test cases based on requirements before writing the actual code. While writing test cases is the centerpiece of TDD, it is time-consuming, expensive, and often shunned by developers. To address these issues associated with TDD, automated test case generation approaches have recently been investigated. Such approaches take source code as input, but not the requirements. Therefore, existing work does not fully support true TDD, as actual code is required to generate test cases. In addition, current deep learning-based test case generation approaches are trained with one learning objective, i.e., to generate test cases that are exactly matched with the ground-truth test cases. However, such approaches may limit the model's ability to generate different yet correct test cases. In this paper, we introduce PyTester, a Text-to-Testcase generation approach that can automatically generate syntactically correct, executable, complete, and effective test cases while being aligned with a given natural language requirement. We evaluate PyTester on the public APPS benchmark dataset, and the results show that our Deep RL approach enables PyTester, a small language model, to outperform much larger language models like GPT3.5, StarCoder, and InCoder. Our findings suggest that future research could consider improving small over large LMs for better resource efficiency by integrating the SE domain knowledge into the design of reinforcement learning architecture.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。