Testing is a key concern when developing process-oriented solutions as it supports modeling experts who have to deal with increasingly complex models and scenarios such as cross-organizational processes. However, the complexity of the research landscape and the diverse set of approaches and goals impedes the analysis and advancement of research and the identification of promising research areas, challenges, and research directions. Hence, a systematic literature review is conducted to identify interesting areas for future research and to provide an overview of existing work. Over 6300 potentially matching publications were determined during the search (literature databases, selected conferences\journals, and snowballing). Finally, 153 publications from 2002 to 2013 were selected, analyzed, and classified. It was found that the software engineering domain has influenced process model testing approaches (e.g., regarding terminology and concepts), but recent publications are presenting independent approaches. Additionally, historical data sources are not exploited to their full potential and current testing related publications frequently contain evaluations of relatively weak quality. Overall, the publication landscape is unevenly distributed so that over 31 publications concentrate on test-case generation but only 4 publications conduct performance test. Hence, the full potential of such insufficiently covered testing areas is not exploited. This systematic review provides a comprehensive overview of the interdisciplinary topic of process model testing. Several open research questions are identified, for example, how to apply testing to cross-organizational or legacy processes and how to adequately include users into the testing methods.
In software testing, the large size of the input domain makes exhaustively testing the inputs a daunting and often impossible task. Pair-wise testing is a popular approach to combinatorial testing problems. This paper reviews Pair-wise testing and its history, strengths, weaknesses, and tools for generating test cases.
Context: Software testing plays an essential role in product quality improvement. For this reason, several software testing models have been developed to support organizations. However, adoption of testing process models inside organizations is still sporadic, with a need for more evidence about reported experiences. Aim: Our goal is to identify results gathered from the application of software testing models in organizational contexts. We focus on characteristics such as the context of use, practices applied in different testing process phases, and reported benefits & drawbacks. Method: We pe…
Context. Multiple automated techniques have been proposed and developed for mobile application GUI testing aiming to improve effectiveness, efficiency, and practicality. The effectiveness, efficiency, and practicality are 3 fundamental characteristics which testing techniques are built upon, and need to be continuously improved to deliver useful solutions for researchers and practitioners, and community as a whole. Objective. In this systematic review, we attempt to provide a broad picture of existing mobile testing tools by collating and analysing their conceptual, and also performance charac…
Revisiting "Revisiting Neuron Coverage for DNN Testing: A Layer-Wise and Distribution-Aware Criterion": A Critical Review and Implications on DNN Coverage Testing
Genetic Algorithms are a popular set of optimization algorithms often used to aid software testing. However, no work has been done to apply systematic software testing techniques to genetic algorithms because of the stochasticity and the lack of known outputs for genetic algorithms. Statistical metamorphic testing is a useful technique for testing programs when the output is unknown or when the program has random elements. In this paper, we identify 17 metamorphic relations for testing a genetic algorithm and show, through mutation testing, that these relations are more effective at finding de…
The requirements in automation, digitalization, and fast computations have loaded the IT sector with expectations of highly reliable, efficient, and cost-effective software. Given that the process of testing, verification, and validation of software products consumes 50-75% of the total revenue if the testing process is ineffective, "n" times the expenditure must be invested to mend the havoc caused. A delay in project completion is often attributed to the testing phase because of the numerous cycles of debugging process. The software testing process determines the face of the product released…
We discuss a range of miscodes found in probabilistic genotyping (PG) software and from other industries that have been reported in the literature and have been used to inform PG admissibility hearings. Every instance of the discovery of a miscode in PG software with which we have been associated has occurred either because of testing, use, or repeat calculation of results either by us or other users. In all cases found during testing or use something has drawn attention to an anomalous result. Intelligent investigation has led to the examination of a small section of the code and detection of…
Context: Context-aware contemporary software systems (CACSS) are mainstream. Furthermore, they present challenges for current engineering practices. These challenges are distinctively present when testing CACSS, as the variation of context deepens the limitations of available software testing practices and technologies. Objective: To understand how the industry deals with the variation of context when testing CACSS. Method: A Rapid Review was commissioned to uncover the necessary evidence to achieve the objectives. Results: Our results show that current research initiatives aim to generate or …
Context: Scientific software plays an important role in critical decision making, for example making weather predictions based on climate models, and computation of evidence for research publications. Recently, scientists have had to retract publications due to errors caused by software faults. Systematic testing can identify such faults in code. Objective: This study aims to identify specific challenges, proposed solutions, and unsolved problems faced when testing scientific software. Method: We conducted a systematic literature survey to identify and analyze relevant literature. We identifie…
Software testing process consists of activities that implemented after it is planned and including to document related testing activities. Test processes must be applied necessarily for able to clearly see the quality of software, the degree of reliability, whether it is ready for delivery, the degree of effectiveness and remains of how much testing. One of the most important phase of these processes is test planning activities. Test planning activities directly affects the project's success in software projects. In this study, software testing process, and test planning activities carried out…
Random testing (RT) is a black-box software testing technique that tests programs by generating random test inputs. It is a widely used technique for software quality assurance, but there has been much debate by practitioners concerning its failure-detection effectiveness. RT is argued to be possibly less effective by some researchers as it does not utilize any information about the program under test. Efforts to mainly improve the failure-detection capability of RT, have led to the proposition of Adaptive Random Testing (ART). ART takes advantage of the location information of previous non-fa…
The emergence of new technologies in software testing has increased the automation and flexibility of the testing process. In this context, the adoption of agents in software testing remains an active research area in which various agent methodologies, architectures, and tools are employed to improve different test problems. Even though research that investigates agents in software testing has been growing, these agent-based techniques should be considered in a broader perspective. In order to provide a comprehensive overview of this research area, which we define as agent-based software testi…
Compilers are widely-used infrastructures in accelerating the software development, and expected to be trustworthy. In the literature, various testing technologies have been proposed to guarantee the quality of compilers. However, there remains an obstacle to comprehensively characterize and understand compiler testing. To overcome this obstacle, we propose a literature analysis framework to gain insights into the compiler testing area. First, we perform an extensive search to construct a dataset related to compiler testing papers. Then, we conduct a bibliometric analysis to analyze the produc…
Digital twins (DT) of industrial processes have become increasingly important. They aim to digitally represent the physical world to help evaluate, optimize, and predict physical processes and behaviors. Therefore, DT is a vital tool to improve production automation through digitalization and becomes more sophisticated due to rapidly evolving simulation and modeling capabilities, integration of IoT sensors with DT, and high-capacity cloud/edge computing infrastructure. However, the fidelity and reliability of DT software are essential to represent the physical world. This paper shows an automa…
Context: With the rising complexity and scale of software systems, there is an ever-increasing demand for sophisticated and cost-effective software testing. To meet such a demand, there is a need for a highly-skilled software testing work-force (test engineers) in the industry. To address that need, many university educators worldwide have included software-testing education in their software engineering (SE) or computer science (CS) programs. Objective: Our objective in this paper is to summarize the body of experience and knowledge in the area of software-testing education to benefit the rea…
Context: The importance of feature modeling languages for software product lines and the planning stage for a systematic literature review. Objective: A protocol for carrying out a systematic literature review about the evidence for identifying and classifying the errors in feature modeling languages. Method: The definition of a protocol to conduct a systematic literature review according to the guidelines of B. Kitchenham. Results: A validated protocol to conduct a systematic literature review. Conclusions: A proposal for the protocol definition of a systematic literature review about the ide…
The recent active development of Internet of Things (IoT) solutions in various domains has led to an increased demand for security, safety, and reliability of these systems. Security and data privacy are currently the most frequently discussed topics; however, other reliability aspects also need to be focused on to maintain the smooth and safe operation of IoT systems. Until now, there has been no systematic mapping study dedicated to the topic of interoperability and integration testing of IoT systems specifically; therefore, we present such an overview in this study. We analyze 803 papers fr…
Context: The importance of defining learning outcomes and the planning stage for a systematic literature review. Objective: A protocol for carrying out a systematic literature review about the evidence for the tool support for the learning outcomes and the teaching-learning process using Bloom's taxonomy to address it. Method: The definition of a protocol to conduct a systematic literature review according to the guidelines of B. Kitchenham. Results: A validated protocol to conduct a systematic literature review. Conclusions: A proposal for the protocol definition of a systematic literature re…
In industry, RESTful APIs are widely used to build modern Cloud Applications. Testing them is challenging, because not only they rely on network communications, but also they deal with external services like databases. Therefore, there has been a large amount of research sprout in recent years on how to automatically verify this kind of web services. In this paper, we review the current state-of-the-art on testing RESTful APIs, based on an analysis of 92 scientific articles. This review categories and summarizes the existing scientific work on this topic, and discusses the current challenges i…
A systematic, language-independent method of finding a minimal set of paths covering the code of a sequential program is proposed for application in White Box testing. Execution of all paths from the set ensures also statement coverage. Execution fault marks problematic areas of the code. The method starts from a UML activity diagram of a program. The diagram is transformed into a directed graph: graph's nodes substitute decision and action points; graph's directed edges substitute action arrows. The number of independent paths equals easy-to-compute cyclomatic complexity of the graph. Associa…
Conducting systematic reviews (SR) is a time consuming endeavor that requires several iterations to setup right. We present ReLiS, a framework to configure and deploy projects while conducting a SR. It features a domain-specific modeling editor tailored for researchers who perform SRs and an architecture that enables live installation and deployment of multiple concurrently running projects. See the accompanying video at http://youtu.be/U5zOmk2vWy8
Lund University（隆德大学）；RISE Research Institutes of Sweden（瑞典RISE研究院）；Axis Communications（安讯士通信公司）
There is a growing interest in industry and academia in machine learning (ML) testing. We believe that industry and academia need to learn together to produce rigorous and relevant knowledge. In this study, we initiate a collaboration between stakeholders from one case company, one research institute, and one university. To establish a common view of the problem domain, we applied an interactive rapid review of the state of the art. Four researchers from Lund University and RISE Research Institutes and four practitioners from Axis Communications reviewed a set of 180 primary studies on ML test…
With the continuous rise in the popularity of Android mobile devices, automated testing of apps has become more important than ever. Android apps are event-driven programs. Unfortunately, generating all possible types of events by interacting with the app's interface is challenging for an automated testing approach. Callback-driven testing eliminates the need for event generation by directly invoking app callbacks. However, existing callback-driven testing techniques assume prior knowledge of Android callbacks, and they rely on a human expert, who is familiar with the Android API, to write stu…
We conduct a systematic literature review on the concept of trust in the worldwide software ecosystem. We acknowledge that trust is something between two actors in the software ecosystem, and we examine what role trust plays in the relationships between end-users and (1) software products, (2) package managers, (3) software producing organizations, and (4) software engineers. Two major findings emerged from the systematic literature review. To begin, we provide a definition of trust in the software ecosystem, including a theoretical framework that decomposes and signifies a theoretical underst…
Context: The Evidence-Based Software Engineering (EBSE) paradigm and the planning phase of a systematic literature review. Objective: A protocol to do a systematic literature review with detailed information about the processes suggested by several guidelines in the field of evidence-based software engineering. Method: An analisys of recent systematic literature reviews published in world leading journals, plus the use of two renowned guidelines and a textbook to sinthetise a formal plan (the protocol). Results: The validated protocol Conclusions: We found that most of the published systematic…
Focus, a framework for formal specification and development of interactive systems, was introduced approx. 25 years ago. Since then this approach was broadly used in academic and industrial studies, as well as provided a basis for a number of another frameworks focusing on particular domains, and for the AF3 modelling tool. In this paper we provide a literature review of the corresponding approaches, academic case studies and industrial applications of these methods.
This technical report presents a Systematic Literature Review (SLR) study that focuses on identifying and classifying the recent research practices pertaining to CPS development through MDE approaches. The study evaluates 140 research papers published during 2010-2018. Accordingly, a comprehensive analysis of various MDE approaches used in the development life-cycle of CPS is presented. Furthermore, the study identifies the research gaps and areas that need more investigation. The contribution helps researchers and practitioners to get an overall understanding of the research trends and existi…
Software Vulnerability Prediction (SVP) is a data-driven technique for software quality assurance that has recently gained considerable attention in the Software Engineering research community. However, the difficulties of preparing Software Vulnerability (SV) related data is considered as the main barrier to industrial adoption of SVP approaches. Given the increasing, but dispersed, literature on this topic, it is needed and timely to systematically select, review, and synthesize the relevant peer-reviewed papers reporting the existing SV data preparation techniques and challenges. We have ca…
Software testing is a critical activity to ensure that software complies with its specification. However, current software testing activities tend not to be completely effective when applied in specific software domains in Virtual Reality (VR) that has several new types of features such as images, sounds, videos, and differentiated interaction, which can become sources of new kinds of faults. This paper presents an overview of the main VR characteristics that can have an impact on verification, validation, and testing (VV&T). Furthermore, it analyzes some of the most successful VR open-source …
Modelling and thus metamodelling have become increasingly important in Software Engineering through the use of Model Driven Engineering. In this paper we present a systematic literature review of instance generation techniques for metamodels, i.e. the process of automatically generating models from a given metamodel. We start by presenting a set of research questions that our review is intended to answer. We then identify the main topics that are related to metamodel instance generation techniques, and use these to initiate our literature search. This search resulted in the identification of 3…
A systematic literature review (SLR) is a methodology used to find and aggregate all relevant existing evidence about a specific research question of interest. Important decisions need to be made at several points in the review process, relating to search of the literature, selection of relevant primary studies and use of methods of synthesis. Visualization can support tasks that involve large collections of data, such as the studies collected, evaluated and summarized in an SLR. The objective of this paper is to present the results of a systematic mapping study (SM) conducted to collect and e…
Context: New scientific evidence continuously arises with advances in Software Engineering (SE) research. Conventionally, Systematic Literature Reviews (SLRs) are not updated or updated intermittently, leaving gaps between updates, during which time the SLR may be missing crucial new evidence. Goal: We propose and evaluate a concept and process called Continuous Systematic Literature Review (CSLR) in SE. Method: To elaborate on the CSLR concept and process, we performed a synthesis of evidence by conducting a meta-ethnography, addressing knowledge from varied research areas. Furthermore, we co…
In this report, we describe the review protocol that will guide the systematic review of the literature in metrics-based discovery of vulnerabilities. The protocol have been developed in adherence with the guidelines for performing Systematic Literature Reviews in Software Engineering prescribed by Kitchenham and Charters.
Mutation analysis assesses a test suite's adequacy by measuring its ability to detect small artificial faults, systematically seeded into the tested program. Mutation analysis is considered one of the strongest test-adequacy criteria. Mutation testing builds on top of mutation analysis and is a testing technique that uses mutants as test goals to create or improve a test suite. Mutation testing has long been considered intractable because the sheer number of mutants that can be created represents an insurmountable problem -- both in terms of human and computational effort. This has hindered th…
The value of a systematic secondary study (a systematic mapping study (SMS) or a systematic literature review (SLR)) comes, directly, from its systematic nature. The formal, well-defined, objective and unbiased process guarantees that the results from these systematically conducted studies are valid. This process is embodied in an action protocol, which must be agreed upon by all the researchers before conducting the secondary study. The protocol is, therefore, a detailed action plan, which contains all the tasks and the ordered sequence of steps to be executed. This document details that prot…
In this paper, we present a tertiary systematic literature review of previous surveys, secondary systematic literature reviews, and systematic mappings. We identify the main observations (what we know) and challenges (what we do not know) on code smells and refactoring. We show that code smells and refactoring have a strong relationship with quality attributes, i.e., with understandability, maintainability, testability, complexity, functionality, and reusability. We argue that code smells and refactoring could be considered as the two faces of a same coin. Besides, we identify how refactoring …
When making choices in software projects, engineers and other stakeholders engage in decision making that involves uncertain future outcomes. Research in psychology, behavioral economics and neuroscience has questioned many of the classical assumptions of how such decisions are made. This literature review aims to characterize the assumptions that underpin the study of these decisions in Software Engineering. We identify empirical research on this subject and analyze how the role of time has been characterized in the study of decision making in SE. The literature review aims to support the dev…
Privacy has been frequently identified as a main concern for system developers while dealing with/managing personal information. Despite this, most existing work on privacy requirements deals with them as a special case of security requirements. Therefore, key aspects of privacy are, usually, overlooked. In this context, wrong design decisions might be made due to insufficient understanding of privacy concerns. In this paper, we address this problem with a systematic literature review whose main purpose is to identify the main concepts/relations for capturing privacy requirements. In addition,…
National Security Institute, Virginia Tech（弗吉尼亚理工学院国家安全研究所）；The MITRE Corporation（米特雷公司）
This paper demonstrates the systematic use of combinatorial coverage for selecting and characterizing test and training sets for machine learning models. The presented work adapts combinatorial interaction testing, which has been successfully leveraged in identifying faults in software testing, to characterize data used in machine learning. The MNIST hand-written digits data is used to demonstrate that combinatorial coverage can be used to select test sets that stress machine learning model performance, to select training sets that lead to robust model performance, and to select data for fine-…
