The requirements in automation, digitalization, and fast computations have loaded the IT sector with expectations of highly reliable, efficient, and cost-effective software. Given that the process of testing, verification, and validation of software products consumes 50-75% of the total revenue if the testing process is ineffective, "n" times the expenditure must be invested to mend the havoc caused. A delay in project completion is often attributed to the testing phase because of the numerous cycles of debugging process. The software testing process determines the face of the product released to the user. It sets the standard and reliability of a company's outputs. As the complexity increases, testing gets intense so as to examine all the outliers and various branches of the processing flow. The testing process is automated using software tools to avoid the tedious manual process of test input generation and validation criteria, which certifies the program only to a certain confidence level in the presence of outliers.
Software testing is a critical element of software quality assurance and represents the ultimate review of specification, design and coding. Software testing is the process of testing the functionality and correctness of software by running it. Software testing is usually performed for one of two reasons: defect detection, and reliability estimation. The problem of applying software testing to defect detection is that software can only suggest the presence of flaws, not their absence (unless the testing is exhaustive). The problem of applying software testing to reliability estimation is that …
As the complexity and scope of game development increase, playtesting remains an essential activity to ensure the quality of video games. Yet, the manual, ad-hoc nature of playtesting gives space to improvements in the process. In this study, we investigate gaps between academic solutions in the literature for automated video game testing and the needs of video game developers in the industry. We performed a literature review on video game automated testing and applied an online survey with video game developers. The literature results show a rise in research topics related to automated video …
Mobile app development involves a unique set of challenges including device fragmentation and rapidly evolving platforms, making testing a difficult task. The design space for a comprehensive mobile testing strategy includes features, inputs, potential contextual app states, and large combinations of devices and underlying platforms. Therefore, automated testing is an essential activity of the development process. However, current state of the art of automated testing tools for mobile apps poses limitations that has driven a preference for manual testing in practice. As of today, there is no c…
Context: Scientific software plays an important role in critical decision making, for example making weather predictions based on climate models, and computation of evidence for research publications. Recently, scientists have had to retract publications due to errors caused by software faults. Systematic testing can identify such faults in code. Objective: This study aims to identify specific challenges, proposed solutions, and unsolved problems faced when testing scientific software. Method: We conducted a systematic literature survey to identify and analyze relevant literature. We identifie…
Background: Research software plays an important role in solving real-life problems, empowering scientific innovations, and handling emergency situations. Therefore, the correctness and trustworthiness of research software are of absolute importance. Software testing is an important activity for identifying problematic code and helping to produce high-quality software. However, testing of research software is difficult due to the complexity of the underlying science, relatively unknown results from scientific algorithms, and the culture of the research software community. Aims: The goal of thi…
Cloud computing is the next stage of the internet evolution. It relies on sharing of resources to achieve coherence on a network. It is emerged as new computing standard that impacts several different research fields, including software testing. There are various software techniques used for testing application. It not only changes the way of obtaining computing resources but also changes the way of managing and delivering computing services, technologies and solutions, meanwhile it causes new issues, challenges and needs in software testing. Software testing in cloud can reduce the need for h…
Automated and therefore repeatable tests are important in product lines to ensure and maintain quality of software functions as well as efficiency of the developers. This article shows methods for fully automated testing of SIMULINK models with the slUnit framework and techniques for the integration of slUnit tests in a general test framework to allow integrated and automated test reports.
Context: With the rising complexity and scale of software systems, there is an ever-increasing demand for sophisticated and cost-effective software testing. To meet such a demand, there is a need for a highly-skilled software testing work-force (test engineers) in the industry. To address that need, many university educators worldwide have included software-testing education in their software engineering (SE) or computer science (CS) programs. Objective: Our objective in this paper is to summarize the body of experience and knowledge in the area of software-testing education to benefit the rea…
The mass production of complex software has made it impossible to manually test it for security vulnerabilities. Automated security testing tools come in a variety of flavors, function at various stages of software development, and target different categories of software vulnerabilities. It is great that we have a plethora of automated tools to choose from, but it is a problem that their adoption and recognition are not prominent. The purpose of this study is to explore the possibilities of existing techniques while also broadening the horizon by exploring the future of security testing tools …
Today's software systems like cyber-physical production systems or big data systems have to process large volumes and diverse types of data which heavily influences the quality of these so-called data-intensive systems. However, traditional software testing approaches rather focus on functional behavior than on data aspects. Therefore, the role of data in testing has to be rethought and specific testing approaches for data-intensive software systems are required. Thus, the aim of this chapter is to contribute to this area by (1) providing basic terminology and background on data-intensive soft…
As software systems are becoming more pervasive, they are also becoming more susceptible to failures, resulting in potentially lethal combinations. Software testing is critical to preventing software failures but is, arguably, the least understood part of the software life cycle and the toughest to perform correctly. Adequate research has been carried out in both the process and technology dimensions of testing, but not in the human dimensions. This paper attempts to fill in the gap by exploring the human dimension, i.e., trying to understand the motivation of software professionals to take up…
The overall aim of the software industry is to ensure delivery of high quality software to the end user. To ensure high quality software, it is required to test software. Testing ensures that software meets user specifications and requirements. However, the field of software testing has a number of underlying issues like effective generation of test cases, prioritisation of test cases etc which need to be tackled. These issues demand on effort, time and cost of the testing. Different techniques and methodologies have been proposed for taking care of these issues. Use of evolutionary algorithms…
Computer games are very challenging to handle for traditional automated testing algorithms. In this paper we will look at intelligent agents as a solution. Agents are suitable for testing games, since they are reactive and able to reason about their environment to decide the action they want to take. This paper presents the experience of using an agent-based automated testing framework called \ivxr\ to test computer games. Three games will be discussed, including a sophisticated 3D game called Space Engineers. We will show how the framework can be used in different ways, either directly to dri…
This study attempts to understand motivators and de-motivators that influence the decisions of software students to take up and sustain software testing careers across three different Asian countries, i.e., China, India, and Malaysia. The re-search question can be framed as How many software students across different Asian geographies are keen to take up testing careers, and what are the reasons for their choices? Towards an answer, we developed a cross-sectional but simple survey-based instrument. In this work, we investigated how software students perceived the software testing role. The res…
This note concerns a search for publications in which the pragmatic concept of a test as conducted in the practice of software testing is formalized, a theory about software testing based on such a formalization is presented or it is demonstrated on the basis of such a theory that there are solid grounds to test software in cases where in principle other forms of analysis could be used. This note reports on the way in which the search has been carried out and the main outcomes of the search. The message of the note is that the fundamentals of software testing are not yet complete in some respe…
The paper presents the experience of the authors in model based testing of safety critical real-time control logic software. It describes specifics of the corresponding industrial settings and discusses technical details of usage of UniTESK model based testing technology in these settings. Finally, we discuss possible future directions of safety critical software development processes and a place of model based testing techniques in it.
Managing large-scale software products is a complex software engineering task. The automation of the software development, release and distribution process is most beneficial in the large collaborations, where the big number of developers, multiple platforms and distributed environment are typical factors. This paper describes Build and Output Analyzer framework and its components that have been developed in CMS to facilitate software maintenance and improve software quality. The system allows to generate, control and analyze various types of automated software builds and tests, such as regula…
Software testing is an important and valuable part of the software development life cycle. Due to time, cost and other circumstances, exhaustive testing is not feasible that's why there is a need to automate the software testing process. Testing effectiveness can be achieved by the State Transition Testing (STT) which is commonly used in real time, embedded and web-based type of software systems. Aim of the current paper is to present an algorithm by applying an ant colony optimization technique, for generation of optimal and minimal test sequences for behavior specification of software. Prese…
Software testing is vital to improve software quality. However, software tester role is stigmatized, partly due to misperception and partly due to the treatment of the testing process within the software industry. The present study analyses this situation aiming to explore what might inhibit an individual from taking up a software testing career. In order to investigate this issue, we surveyed 132 senior students pursuing degrees in information systems, information and communication technology, computer science, computer engineering, software engineering, and other closely-related disciplines …
A program fails. Under which circumstances does this failure occur? One single algorithm, the delta debugging algorithm, suffices to determine these failure-inducing circumstances. Delta debugging tests a program systematically and automatically to isolate failure-inducing circumstances such as the program input, changes to the program code, or executed statements.
To learn how to introduce automated regression testing to existing medium scale Open Source projects, a long-term field experiment was performed with the Open Source project FreeCol. Results indicate that (1) introducing testing is both beneficial for the project and feasible for an outside innovator, (2) testing can enhance communication between developers, (3) signaling is important for engaging the project participants to fill a newly vacant position left by a withdrawal of the innovator. Five prescriptive strategies are extracted for the innovator and two conjectures offered about the abil…
This paper attempts to understand motivators and de-motivators that influence the decisions of software professionals to take up and sustain software testing careers across four different countries, i.e. Canada, China, Cuba, and India. The research question can be framed as "How many software professionals across different geographies are keen to take up testing careers, and what are the reasons for their choices?" Towards that, we developed a cross-sectional but simple survey-based instrument. In this study we investigated how software testers perceived and valued what they do and their envir…
Software testing uses wide range of different tools to enhance the complicated process of defining quality of the system under test. Formal Concept Analysis (FCA) provides us with algorithms of deriving formal ontology from a set of objects and their attributes. With the use of FCA we can considerably improve the efficiency of test case derivation. Moreover, an FCA-based machine learning system supports the analysis of regression testing results.
Software testing is one of the crucial supporting processes of software life cycle. Unfortunately for the software industry, the role is stigmatized, partly due to misperception and partly due to treatment of the role in the software industry. The present study aims to analyse this situation to explore what inhibit an individual from taking up a software testing career. In order to investigate this issue, we surveyed 82 senior students taking a degree in information technology, information and communication technology, and computer science at two Malaysian universities. The subjects were asked…
Context: While many companies conduct their software testing activities in-house, many other companies outsource their software testing needs to other firms who act as software testing service providers. As a result, Testing as a Service (TaaS) has emerged as a strong service industry in the last several decades. In the context of software testing services, there could be various challenges (e.g., during the planning and service delivery phases) and, as a result, the quality of testing services is not always as expected. Objective: It is important, for both providers and also customers of test…
Context: Software testing plays an essential role in product quality improvement. For this reason, several software testing models have been developed to support organizations. However, adoption of testing process models inside organizations is still sporadic, with a need for more evidence about reported experiences. Aim: Our goal is to identify results gathered from the application of software testing models in organizational contexts. We focus on characteristics such as the context of use, practices applied in different testing process phases, and reported benefits & drawbacks. Method: We pe…
This paper is about understanding the nature of bug fixing by analyzing thousands of bug fix transactions of software repositories. It then places this learned knowledge in the context of automated program repair. We give extensive empirical results on the nature of human bug fixes at a large scale and a fine granularity with abstract syntax tree differencing. We set up mathematical reasoning on the search space of automated repair and the time to navigate through it. By applying our method on 14 repositories of Java software and 89,993 versioning transactions, we show that not all probabilist…
Ensuring the quality of automated driving systems is a major challenge the automotive industry is facing. In this context, quality defines the degree to which an object meets expectations and requirements. Especially, automated vehicles at SAE level 4 and 5 will be expected to operate safely in various contexts and complex situations without misconduct. Thus, a systematic approach is needed to show their safe operation. A way to address this challenge is simulation-based testing as pure physical testing is not feasible. During simulation-based testing, the data used to evaluate the actual qual…
Research software is often developed by individual researchers or small teams in parallel to their research work. The more people and research projects rely on the software in question, the more important it is that software updates implement new features correctly and do not introduce regressions. Thus, developers of research software must balance their limited resources between implementing new features and thoroughly testing any code changes. We present the processes we use for developing the distributed integration framework RCE at DLR. These processes aim to strike a balance between autom…
Many scientific-software projects test their codes inadequately, or not at all. Despite its well-known benefits, adopting routine testing is often not easy. Development teams may have doubts about establishing effective test procedures, writing test software, or handling the ever-growing complexity of test cases. They may need to run (and test) on restrictive HPC platforms. They almost certainly face time and budget pressures that can keep testing languishing near the bottom of their to-do lists. This paper presents DDTS, a framework for building test suite applications, designed to fit scient…
This preprint specifies supplementary material and the results of a comparative, syntactic, and semantic study of terms for three software testing glossaries. The three software testing glossaries are: the ISO 29119-1, Concepts and definitions standard, the Standard Glossary of Terms used in Software Testing (version 3.5) by the International Software Testing Qualifications Board (ISTQB), and the glossary provided by the Test Maturity Model integration (TMMi) Foundation. First, we categorized the glossaries terms by analyzing their given semantics into 8 categories, namely: test project, strat…
This paper presents the core principles of reliability in software engineering - outlining why reliability testing is critical and specifying the process of measuring reliability. The paper provides insight for both novice and experts in the software engineering field for assessing failure intensity as well as predicting failure of software systems. Measurements are conducted by utilizing information from an operational profile to further enhance a test plan and test cases, all of which this paper demonstrates how to implement.
Search-based software testing (SBT) is an effective and efficient approach for testing automated driving systems (ADS). However, testing pipelines for ADS testing are particularly challenging as they involve integrating complex driving simulation platforms and establishing communication protocols and APIs with the desired search algorithm. This complexity prevents a wide adoption of SBT and thorough empirical comparative experiments with different simulators and search approaches. We present OpenSBT, an open-source, modular and extensible framework to facilitate the SBT of ADS. With OpenSBT, i…
In software development process we come across various modules. Which raise the idea of priority of the different modules of a software so that important modules are tested on preference. This approach is desirable because it is not possible to test each module regressively due to time and cost constraints. This paper discuss on some parameters, required to prioritize several modules of a software and provides measure of optimal time and cost for testing based on non homogeneous Poisson process.
Advancing ocean science has a significant impact to the development of the world, from operating a safe navigation for vessels to maintaining a healthy and diverse ocean ecosystem. Various ocean software systems have been extensively adopted for different purposes, for instance, predicting hourly sea level elevation across shorelines, simulating large-scale ocean circulations, as well as integrating into Earth system models for weather forecasts and climate projections. Regardless of their significance, guaranteeing the trustworthiness of ocean software and modelling systems is a long-standing…
Service oriented architecture (SOA) is one of the latest software architectures. This architecture is created in direction of the business requirements and removed the gap between softwares and businesses. The software testing is the rising cost of activities in development software. SOA has different specifications and features proportion of the other software architectures. First this paper reviews SOA testing challenges and existing solution(s) for those challenges. Then that reports a survey of recent research to SOA systems testing, that covers both functional and non-functional testing. …
Many software developments projects fail due to quality problems. Software testing enables the creation of high quality software products. Since it is a cumbersome and expensive task, and often hard to manage, both its technical background and its organizational implementation have to be well founded. We worked with regional companies that develop software in order to learn about their distinct weaknesses and strengths with regard to testing. Analyzing and comparing the strengths, we derived best practices. In this paper we explain the project's background and sketch the design science researc…
Quality assurance makes sure the project will be completed based on the previously approved specifications, standards and functionality. It is required without defects and possible problems. It monitors and tries to progress the development process from the start of the project. Software Quality Assurance (SQA) is the combination of the entire software development process, which includes software design, coding, source code control, code review, change management, configuration management and release management. In this paper we describe the solution for the key problems of software testing in…
The emergence of new technologies in software testing has increased the automation and flexibility of the testing process. In this context, the adoption of agents in software testing remains an active research area in which various agent methodologies, architectures, and tools are employed to improve different test problems. Even though research that investigates agents in software testing has been growing, these agent-based techniques should be considered in a broader perspective. In order to provide a comprehensive overview of this research area, which we define as agent-based software testi…
Software testing process consists of activities that implemented after it is planned and including to document related testing activities. Test processes must be applied necessarily for able to clearly see the quality of software, the degree of reliability, whether it is ready for delivery, the degree of effectiveness and remains of how much testing. One of the most important phase of these processes is test planning activities. Test planning activities directly affects the project's success in software projects. In this study, software testing process, and test planning activities carried out…
Context. Recently, a family of studies highlighted the unpopularity of software testing careers among undergraduate students in software engineering and computer science courses. The original study and its replications explored the perception of students in universities in four countries (Cana-da, China, India, and Malaysia), and indicated that most students do not consider a career in software testing as an option after graduation. This scenario represents a problem for the software industry since the lack of skilled testing professionals might decrease the quality of software projects and in…
