Constraints among test parameters often have substantial effects on the performance of test case generation for combinatorial interaction testing. This paper investigates the effectiveness of the use of Binary Decision Diagrams (BDDs) for constraint handling. BDDs are a data structure used to represent and manipulate Boolean functions. The core role of a constraint handler is to perform a check to determine if a partial test case with unspecified parameter values satisfies the constraints. In the course of generating a test suite, this check is executed a number of times; thus the efficiency of the check significantly affects the overall time required for test case generation. In the paper, we study two different approaches. The first approach performs this check by computing the logical AND of Boolean functions that represent all constraint-satisfying full test cases and a given partial test case. The second approach uses a new technique to construct a BDD that represents all constraint-satisfying partial test cases. With this BDD, the check can be performed by simply traversing the BDD from the root to a sink. We developed a program that incorporates both approaches into IPOG, a well-known test case generation algorithm. Using this program, we empirically evaluate the performance of these BDD-based constraint handling approaches using a total of 62 problem instances. In the evaluation, the two approaches are compared with three different constraint handling approaches, namely, those based on Boolean satisfiability (SAT) solving, Minimum Forbidden Tuples (MFTs), and Constraint Satisfiction Problem (CSP) solving. The results of the evaluation show that the two BDD-based approaches usually outperform the other constraint handling techniques and that the BDD-based approach using the new technique exhibits best performance.
An Extensive Empirical Study on Evaluation Metrics for Combinatorial Interaction Testing
School of Computer Science and Engineering, Macau University of Science and Technology(澳门科技大学计算机科学与工程学院);Macau University of Science and Technology Zhuhai MUST Science and Technology Research Institute(澳门科技大学珠海澳门科技大学研究院);School of Information Science and Technology, Dalian Maritime University(大连海事大学信息科学与技术学院);School of Science, University of Tokyo(东京大学理学部);Department of Electrical and Computer Engineering, University of Alberta(阿尔伯塔大学电气与计算机工程系)
Harbin Institute of Technology(哈尔滨工业大学);Nanyang Technological University(南洋理工大学);New York University Shanghai(上海纽约大学);University of Illinois at Urbana–Champaign(伊利诺伊大学厄巴纳-香槟分校);Kyushu University(九州大学)
Supplementary Material for the Information Sciences Paper: An Experimental Study of Hyper-Heuristic Selection and Acceptance Mechanism for Combinatorial t-way Test Suite Generation
Time-Space Efficient Regression Testing for Configurable Systems
可配置系统是指可以从一组选项中进行调整的系统。它们是普遍存在的,测试它们是重要和具有挑战性的。用于测试可配置系统的现有方法要么是不可靠的(即,它们可能错过断层揭示构造)或不成比例。本文提出了EvoSPLat,可配置系统的回归测试技术。EvoSPLat建立在我们以前开发的技术SPLat的基础上,SPLat可以从测试中探索所有动态可达配置。EvoSPLat针对回归测试中的两种使用场景进行了调优:回归配置选择(RCS)和回归测试选择(RTS)。EvoSPLat for RCS修剪不受更改影响的配置(非测试),而EvoSPLat for RTS修剪不受更改影响的测试(非配置)。在进化的背景下处理这两种情况是很重要的。实验结果表明,EvoSPLat是有前途的。我们观察到可配置Java程序的时间(22%)和配置数量(45%)大幅减少。在一个大型的真实世界可配置系统(GCC)的案例研究中,EvoSPLat减少了35%的运行时间。将EvoSPLat与采样技术进行比较,2-wise是最有效的技术,但它错过了两个错误,而EvoSPLat检测所有错误的速度平均比6-wise快四倍。
An Evaluation of Monte Carlo-Based Hyper-Heuristic for Interaction Testing of Industrial Embedded Software Applications
超启发式算法是一种新的自适应混合元启发式算法,以获得一个通用的算法来解决优化问题。这项工作的重点是选择类型的超启发式,称为指数蒙特卡罗计数器(EMCQ)。当前实现依赖于无记忆选择,这可能是适得其反的,因为所选择的搜索运算符可能(历史上)不是当前搜索实例的最佳执行运算符。为了解决这个问题,我们建议将内存集成到EMCQ中,使用基于Q学习机制的强化学习(称为Q-EMCQ)进行组合t-wise测试集生成。组合测试生成在工业程序中的有限应用可能会影响Q-EMCQ等技术的使用。因此,有必要针对相关的工业软件评估这种方法,目的是显示覆盖代码以及查找错误所需的交互程度。我们将Q-EMCQ应用于37个用功能框图(FBD)语言编写的实际工业程序,这些程序用于在Bombardier Transportation Sweden AB开发训练控制管理系统。研究结果表明,Q-EMCQ是一种有效的测试用例生成技术。此外,与处理最小化问题的t-明智测试集生成不同,我们还将Q-EMCQ进行了涉及一般模块聚类的最大化问题,以证明我们方法的有效性。
ReCatcher: Towards LLMs Regression Testing for Code Generation
Huawei Distributed Scheduling and Data Engine Lab(华为分布式调度与数据引擎实验室);Huawei Distributed Scheduling(华为分布式调度);Data Engine Lab Montreal Canada(数据引擎实验室蒙特利尔加拿大);Data Engine Lab(数据引擎实验室)