Evaluating the behavioral boundaries of deep learning (DL) systems is crucial for understanding their reliability across diverse, unseen inputs. Existing solutions fall short as they rely on untargeted random, model- or latent-based perturbations, due to difficulties in generating controlled input variations. In this work, we introduce Mimicry, a novel black-box test generator for fine-grained, targeted exploration of DL system boundaries. Mimicry performs boundary testing by leveraging the probabilistic nature of DL outputs to identify promising directions for exploration. It uses style-based GANs to disentangle input representations into content and style components, enabling controlled feature mixing to approximate the decision boundary. We evaluated Mimicry's effectiveness in generating boundary inputs for five widely used DL image classification systems of increasing complexity, comparing it to two baseline approaches. Our results show that Mimicry consistently identifies inputs closer to the decision boundary. It generates semantically meaningful boundary test cases that reveal new functional (mis)behaviors, while the baselines produce mainly corrupted or invalid inputs. Thanks to its enhanced control over latent space manipulations, Mimicry remains effective as dataset complexity increases, maintaining competitive diversity and higher validity rates, confirmed by human assessors.
Department of Philosophy（哲学系）；Carnegie Mellon University（卡内基梅隆大学）；Department of Psychiatry & Behavioral Sciences（精神病学与行为科学系）；University of Minnesota（明尼苏达大学）
Variational autoencoder (VAE) neural networks can be trained to generate power system states that capture both marginal distribution and multivariate dependencies of historical data. The coordinates of the latent space codes of VAEs have been shown to correlate with conceptual features of the data, which can be leveraged to synthesize targeted data with desired features. However, the locations of the VAEs' latent space codes that correspond to specific properties are not constrained. Additionally, the generation of data with specific characteristics may require data with corresponding hard-to-…
Safety is an essential aspect in the facilitation of automated vehicle deployment. Current testing practices are not enough, and going beyond them leads to infeasible testing requirements, such as needing to drive billions of kilometres on public roads. Automated vehicles are exposed to an indefinite number of scenarios. Handling of the most challenging scenarios should be tested, which leads to the question of how such corner cases can be determined. We propose an approach to identify the performance boundary, where these corner cases are located, using Gaussian Process Classification. We als…
The research on developing software defect prediction (SDP) models is targeted at reducing the workload on the tester and, thereby, the time spent on the targeted module. However, while a considerable amount of research has been done on developing prediction models or attempting to mitigate the related issues in developing prediction models, it is still unknown whether the developed prediction model really works in real-world testing environments or not. With this article, we bridge this research gap of finding the feasibility of the developed binary defect prediction model in the real-world t…
ResTest: Targeted Coverage of Residual Not-Covered Code Using Large Language Models for Web GUI Testing
Key Lab of HCST (PKU), MOE（北京大学）；SCS（北京大学）；Peking University, Beijing, China（中国科学院软件研究所；中国科学院大学）；Key Lab of HCST (PKU) & NERC of SE, MOE（北京通明湖信息技术应用创新中心）；Peking University, Beijing, China（复旦大学高级计算系统研究所）；Institute of Software Chinese Academy of Sciences, University of Chinese Academy of Sciences, Beijing, China；Beijing Tongming Lake Information Technology Application Innovation Center (TLAIC), China；Fudan University Institute of Systems for Advanced Computing, China
Testing has been widely recognised as difficult for AI applications. This paper proposes a set of testing strategies for testing machine learning applications in the framework of the datamorphism testing methodology. In these strategies, testing aims at exploring the data space of a classification or clustering application to discover the boundaries between classes that the machine learning application defines. This enables the tester to understand precisely the behaviour and function of the software under test. In the paper, three variants of exploratory strategies are presented with the algo…
Volvo Group（沃尔沃集团）；Chalmers University of Technology（查尔姆斯理工大学）
Automotive software testing continues to rely largely upon expensive field tests to ensure quality because alternatives like simulation-based testing are relatively immature. As a step towards lowering reliance on field tests, we present SilGAN, a deep generative model that eases specification, stimulus generation, and automation of automotive software-in-the-loop testing. The model is trained using data recorded from vehicles in the field. Upon training, the model uses a concise specification for a driving scenario to generate realistic vehicle state transitions that can occur during such a s…
A natural and often used strategy when testing software is to use input values at boundaries, i.e. where behavior is expected to change the most, an approach often called boundary value testing or analysis (BVA). Even though this has been a key testing idea for long it has been hard to clearly define and formalize. Consequently, it has also been hard to automate. In this research note we propose one such formalization of BVA by, in a similar way as to how the derivative of a function is defined in mathematics, considering (software) program derivatives. Critical to our definition is the notion…
We elaborate on a metamorphic approach for testing industrial control systems. The proposed approach consists of two phases: an exploration phase in which we learn about fault patterns of the system under test and an exploitation phase where the observed fault patterns are used for targeted testing. Our method extracts metamorphic relations and input space of the system from its requirements. The seed input used for testing is extracted from the execution logs of the system and used to generate source tests and follow-up tests automatically. The morphed input is constructed based on the seed i…
Graz University of Technology（格拉茨工业大学）；Silicon Austria Labs（奥地利硅实验室）；Lamarr Security Research（拉马尔安全研究中心）
Evaluation of deep reinforcement learning (RL) is inherently challenging. Especially the opaqueness of learned policies and the stochastic nature of both agents and environments make testing the behavior of deep RL agents difficult. We present a search-based testing framework that enables a wide range of novel analysis capabilities for evaluating the safety and performance of deep RL agents. For safety testing, our framework utilizes a search algorithm that searches for a reference trace that solves the RL task. The backtracking states of the search, called boundary states, pose safety-critica…
Mining Workflow Graphs for Black-Box Boundary Testing of Conversational LLM Agents
Lero, the Research Ireland Centre for Software University of Limerick（利罗爱尔兰软件研究中心，利默里克大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；University of Ottawa（渥太华大学）；Genesys Ireland
Failure region, where failure-causing inputs reside, has provided many insights to enhance testing effectiveness of many testing methods. Failure region may also provide some important information to support other processes such as software debugging. When a testing method detects a software failure, indicating that a failure-causing input is identified, the next important question is about how to identify the failure region based on this failure-causing input, i.e., Identification of Failure Regions (IFR). In this paper, we introduce a new IFR strategy, namely Search for Boundary (SB), to ide…
Constrained Co-evolutionary Metamorphic Differential Testing for Autonomous Systems with an Interpretability Approach
School of EECS, University of Ottawa（渥太华大学电气与计算机工程学院）；Lero SFI centre for Software Research, University of Limerick（利默里克大学Lero软件研究中心）；Waterloo Research Center of Huawei Technologies Canada Co., Ltd.（华为技术有限公司加拿大分公司滑铁卢研发中心）
For software to be reliable and resilient, it is widely accepted that tests must be created and maintained alongside the software itself. One safeguard from vulnerabilities and failures in code is to ensure correct behavior on the boundaries between the input space sub-domains. So-called boundary value analysis (BVA) and boundary value testing (BVT) techniques aim to exercise those boundaries and increase test effectiveness. However, the concepts of BVA and BVT themselves are not generally well defined, and it is not clear how to identify relevant sub-domains, and thus the boundaries delineati…
This report is targeted to groups who are subject matter experts in their application but deep learning novices. It contains practical advice for those interested in testing the use of deep neural networks on applications that are novel for deep learning. We suggest making your project more manageable by dividing it into phases. For each phase this report contains numerous recommendations and insights to assist novice practitioners.
AdaT$^2$: Adaptive Test Transformations for Black-Box Boundary Testing of Conversational Agents
Lero, the Research Ireland Centre for Software, University of Limerick（利默里克大学Lero爱尔兰软件研究中心）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；University of Ottawa（渥太华大学）；Genesys（Genesys公司）
Infotiv AB（英孚提夫公司）；RISE Research Institutes of Sweden（瑞典RISE研究院）；University of Gothenburg（哥德堡大学）；Lund University（隆德大学）
With the growing capabilities of autonomous vehicles, there is a higher demand for sophisticated and pragmatic quality assurance approaches for machine learning-enabled systems in the automotive AI context. The use of simulation-based prototyping platforms provides the possibility for early-stage testing, enabling inexpensive testing and the ability to capture critical corner-case test scenarios. Simulation-based testing properly complements conventional on-road testing. However, due to the large space of test input parameters in these systems, the efficient generation of effective test scenar…
Tencent Inc.（腾讯公司）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Texas at Dallas（德克萨斯大学达拉斯分校）；ETH Zurich（苏黎世联邦理工学院）
Neural Machine Translation (NMT) has been widely adopted recently due to its advantages compared with the traditional Statistical Machine Translation (SMT). However, an NMT system still often produces translation failures due to the complexity of natural language and sophistication in designing neural networks. While in-house black-box system testing based on reference translations (i.e., examples of valid translations) has been a common practice for NMT quality assurance, an increasingly critical industrial practice, named in-vivo testing, exposes unseen types or instances of translation fail…
