The automated generation of test code can reduce the time and effort required to build software while increasing its correctness and robustness. In this paper, we present RE-ASSERT, an approach for the automated generation of JUnit test asserts which produces more accurate asserts than previous work with fewer constraints. This is achieved by targeting projects individually, using precise code-to-test traceability for learning and by generating assert statements from the method-under-test directly without the need to write an assert-less test first. We also utilise Reformer, a state-of-the-art deep learning model, along with two models from previous work to evaluate ReAssert and an existing approach, known as ATLAS, using lexical accuracy,uniqueness, and dynamic analysis. Our evaluation of ReAssert shows up to 44% of generated asserts for a single project match exactly with the ground truth, increasing to 51% for generated asserts that compile. We also improve on the ATLAS results through our use of Reformer with 28% of generated asserts matching exactly with the ground truth. Reformer also produces the greatest proportion of unique asserts (71%), giving further evidence that Reformer produces the most useful asserts.
AutoAssert 1: A LoRA Fine-Tuned LLM Model for Efficient Automated Assertion Generation
Institute of Computing Technology（计算技术研究所）；Chinese Academy of Sciences（中国科学院）；International College（国际学院）；Beijing University of Posts and Telecommunications（北京邮电大学）；University of Chinese Academy of Sciences（中国科学院大学）
Test case generation is an important activity, yet a time-consuming and laborious task. Recently, AthenaTest -- a deep learning approach for generating unit test cases -- is proposed. However, AthenaTest can generate less than one-fifth of the test cases correctly, due to a lack of assertion knowledge and test signature verification. In this paper, we propose A3Test, a DL-based test case generation approach that is augmented by assertion knowledge with a mechanism to verify naming consistency and test signatures. A3Test leverages the domain adaptation principles where the goal is to adapt the …
Functional validation is necessary to detect any errors during quantum computation. There are promising avenues to debug quantum circuits using runtime assertions. However, the existing approaches rely on the expertise of the verification engineers to manually design and insert the assertions in suitable locations. In this paper, we propose automated generation and placement of quantum assertions based on static analysis and random sampling of quantum circuits. Specifically, this paper makes two important contributions. We automatically uncover special properties of a quantum circuit, such as …
Software testing is an essential part of the software lifecycle and requires a substantial amount of time and effort. It has been estimated that software developers spend close to 50% of their time on testing the code they write. For these reasons, a long standing goal within the research community is to (partially) automate software testing. While several techniques and tools have been proposed to automatically generate test methods, recent work has criticized the quality and usefulness of the assert statements they generate. Therefore, we employ a Neural Machine Translation (NMT) based appro…
Improving Dynamic Specification Inference with LLM-Generated Counterexamples
University of Rio Cuarto（里奥夸尔托大学）；Complutense University of Madrid（马德里康普顿斯大学）；University of Buenos Aires（布宜诺斯艾利斯大学）；Luxembourg Institute of Science and Technology（卢森堡科学与技术研究所）；National Council for Scientific and Technical Research (CONICET)（国家科学研究和技术委员会）；Guangdong Technion-Israel Institute of Technology（广东以色列理工学院）
Various mature automated test generation tools exist for statically typed programming languages such as Java. Automatically generating unit tests for dynamically typed programming languages such as Python, however, is substantially more difficult due to the dynamic nature of these languages as well as the lack of type information. Our Pynguin framework provides automated unit test generation for Python. In this paper, we extend our previous work on Pynguin to support more aspects of the Python language, and by studying a larger variety of well-established state of the art test-generation algor…
SPARC: Scenario Planning and Reasoning for Automated C Unit Test Generation
Bangladesh University of Engineering and Technology（孟加拉工程科技大学）；National Taiwan University（台湾国立大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Testing is widely recognized as an important stage of the software development lifecycle. Effective software testing can provide benefits such as bug finding, preventing regressions, and documentation. In terms of documentation, unit tests express a unit's intended functionality, as conceived by the developer. A test oracle, typically expressed as an condition, documents the intended behavior of a unit under a given test prefix. Synthesizing a functional test oracle is a challenging problem, as it must capture the intended functionality rather than the implemented functionality. In this paper,…
LLM-Based Test Generation: Information Sources, Generation Strategies, and Quality Evidence
Chengdu Institute of Computer Applications, Chinese Academy of Sciences（中国科学院成都计算机应用研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Institute of Multidisciplinary Research for Advanced Materials (IMRAM), Tohoku University（东北大学先进材料多学科研究所）；Artificial Intelligence Research Institute, Shenzhen University of Advanced Technology（深圳先进技术研究院人工智能研究所）
Branch coverage of source code is a very widely used test criterion. Moreover, branch coverage is a similar problem to line coverage, MC/DC and the coverage of assertion violations, certain runtime errors and various other types of test objective. Indeed, establishing that a large number of test objectives are unreachable, or conversely, providing the test inputs which reach them, is at the heart of many verification tasks. However, automatic test generation for exhaustive branch coverage remains an elusive goal: many modern tools obtain high coverage scores without being able to provide an ex…
Veri-Sure: Multi-Agent RTL Code Generation with Temporal Tracing, Slicing and Formal Verification
School of Physics and Astronomy, The University of Edinburgh, Edinburgh, UK（物理与天文学院，爱丁堡大学，爱丁堡，英国）；State Key Laboratory of Analog and Mixed-Signal VLSI, University of Macau, Macau（模拟与混合信号VLSI国家重点实验室，澳门大学，澳门）；School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen, Shenzhen, China（科学与工程学院，香港中文大学（深圳））
Escaping the Self-Repair Trap: Improving Test Oracle Generation via Dual-Context Awareness
Beihang University（北京航空航天大学）；National College for Excellent Engineers（卓越工程师学院）；Qingdao Research Institute（青岛研究院）；Hangzhou Innovation Institute（杭州创新研究院）
