The popularity of continuous integration (CI) is increasing as a result of market pressure to release product features or updates frequently. The ability of CI to deliver quality at speed depends on reliable test automation. In this paper, we present an empirical study to observe the effect of test automation maturity (assessed by standard best practices in the literature) on product quality, test automation effort, and release cycle in the CI context of open source projects. We run our test automation maturity survey and got responses from 37 open source java projects. We also mined software repositories of the same projects. The main results of regression analysis reveal that, higher levels of test automation maturity are positively associated with higher product quality (p-value=0.000624) and shorter release cycle (p-value=0.01891); There is no statistically significant evidence of increased test automation effort due to higher levels of test automation maturity and product quality. Thus, we conclude that, a potential benefit of improving test automation maturity (using standard best practices) is product quality improvement and release cycle acceleration in the CI context of open source projects. We encourage future research to extend our findings by adding more datasets with different programming languages and CI tools, closed source projects, and large-scale industrial projects. Our recommendation to practitioners (in the similar CI context) is to utilize standard best practices to improve test automation maturity.
The software industry has seen an increasing interest in test automation. In this paper, we present a test automation maturity survey serving as a self-assessment for practitioners. Based on responses of 151 practitioners coming from above 101 organizations in 25 countries, we make observations regarding the state of the practice of test automation maturity: a) The level of test automation maturity in different organizations is differentiated by the practices they adopt; b) Practitioner reported the quite diverse situation with respect to different practices, e.g., 85\% practitioners agreed th…
Test automation is important in software industry but self-assessment instruments for assessing its maturity are not sufficient. The two objectives of this study are to synthesize what an organization should focus to assess its test automation; develop a self-assessment instrument (a survey) for assessing test automation maturity and scientifically evaluate it. We carried out the study in four stages. First, a literature review of 25 sources was conducted. Second, the initial instrument was developed. Third, seven experts from five companies evaluated the initial instrument. Content Validity I…
Mature test automation is key for achieving software quality at speed. In this paper, we present a multivocal literature review with the objective to survey and synthesize the guidelines given in the literature for improving test automation maturity. We selected and reviewed 81 primary studies, consisting of 26 academic literature and 55 grey literature sources. From primary studies, we extracted 26 test automation best practices (e.g., Define an effective test automation strategy, Set up good test environments, Develop high-quality test scripts) and collected many pieces of advice (e.g., in f…
How to successfully conduct test automation process improvement (TAPI) for continuous development, consisting of iterative software development, continuous testing, and delivery, is the challenge faced by many software organizations. In this paper, we present an experience report on TAPI in one DevOps team in F-Secure (a Finnish software company). The team builds Windows application software and exists in F-Secure's TAPI culture. The team self-reports high satisfaction and maturity in test automation for continuous development. To study their TAPI, we reviewed a collection of experience notes,…
GÉANT project is an example of a large organization with around 30 software projects and around 20 software development teams. Software development teams consist of many skilled associates coming from all members National Research and Education Networks. Three main issues that are common for all these software development teams and their members are: geographical distribution, scattered manpower percentage, and parallel involvement in other high priority projects in their native organizations. This paper presents a novel software maturity model that is designed specifically for GÉANT software …
Sopra Steria（索拉斯特里亚公司）；Université de Lorraine（洛林大学）；IAE Lyon School of Management（里昂管理学院（IAE））
The project management field has the imperative to increase the project probability of success. Experts have developed several project management maturity models to assets and improve the project outcome. However, the current literature lacks of models allowing correlating the measured maturity and the expected probability of success. This paper uses the characteristics of Bayesian networks to formalize experts' knowledge and to extract knowledge from a project overcost database. It develops a method to estimate the impact of project management maturity on the risk of project overcost. A gener…
With the agile approach to managing software development projects comes an increased dependability on well functioning teams, since many of the practices are built on teamwork. The objective of this study was to investigate if, and how, team development from a group psychological perspective is related to some work practices of agile teams. Data were collected from 34 agile teams (200 individuals) from six software development organizations and one university in both Brazil and Sweden using the Group Development Questionnaire (Scale IV) and the Perceptive Agile Measurement (PAM). The result in…
The agile approach to projects focuses more on close-knit teams than traditional waterfall projects, which means that aspects of group maturity become even more important. This psychological aspect is not much researched in connection to the building of an "agile team." The purpose of this study is to investigate how building agile teams is connected to a group development model taken from social psychology. We conducted ten semi-structured interviews with coaches, Scrum Masters, and managers responsible for the agile process from seven different companies, and collected survey data from 66 gr…
Defect estimation and prediction are some of the main modulating factors for the success of software projects in any software industry. Maturity and competency of a project manager in efficient prediction and estimation of resource capabilities are one of the strategic driving forces towards the generation of high quality software. Currently, there are no estimation techniques developed through empirical analysis to evaluate the decision capability of a project manager towards resource allocation for effective defect management. This paper brings out an empirical study carried out in a product…
Automated testing improves the efficiency of testing practice at various levels of projects in the organization. Unfortunately, we do not have a common architecture or common standards for designing frameworks across different test levels, projects and test tools which can assist developers, testers and business analysts. To address the above problem, in this paper, I have first proposed a unique reference model and then a design architecture using the proposed model for designing any Data Driven Automation Frameworks. The reference model is K model which can be used for modeling any data-driv…
Context: User-Centered Design and Agile methodologies focus on human issues. Nevertheless, agile methodologies focus on contact with contracting customers and generating value for them. Usually, the communication between end users and the agile team is mediated by customers. However, they do not know the problems end users face in their routines. Hence, UX issues are typically identified only after the implementation, during user testing and validation. Objective: Aiming to improve the understanding and definition of the problem in agile projects, this research investigates the practices and d…
The Agentic Automation Canvas: a structured framework for agentic AI project design
Institute of Computational Biology, Computational Health Center, Helmholtz Center, Munich, Germany（慕尼黑亥姆霍兹中心计算生物学研究所，计算健康中心）；German Center for Diabetes Research, Munich, Germany（德国糖尿病研究中心）；Open Targets, European Bioinformatics Institute (EMBL-EBI), Hinxton, Cambridge, United Kingdom（开放目标计划，欧洲生物信息学研究所（EMBL-EBI））
Bots are becoming a popular method for automating basic everyday tasks in many software projects. This is true in particular because of the availability of many off-the-shelf task-specific bots that teams can quickly adopt (which are sometimes completed with additional task-specific custom bots). Based on our experience in the Coq project, where we have developed and maintained a multi-task project-specific bot, we argue that this alternative approach to project automation should receive more attention because it strikes a good balance between productivity and adaptibility. In this article, we…
Digital solutions have substantially contributed to the growth and dissemination of education. The distance education modality has been presented as an opportunity for worldwide students in many types of courses. However, projects of digital educational platforms require different expertise including knowledge areas such as pedagogy, psychology, computing, and digital technologies associated with education that allow the correct development and application of these solutions. To support the evolution of such solutions with satisfactory quality indicators, this research presents a model focused…
Marketplaces for distributing software products and services have been getting increasing popularity. GitHub, which is most known for its version control functionality through Git, launched its own marketplace in 2017. GitHub Marketplace hosts third party apps and actions to automate workflows in software teams. Currently, this marketplace hosts 440 Apps and 7,878 Actions across 32 different categories. Overall, 419 Third party developers released their apps on this platform which 111 distinct customers adopted. The popularity and accessibility of GitHub projects have made this platform and th…
Distributions of open source software packages dedicated to specific programming languages facilitate software development by allowing software projects to depend on the functionality provided by such reusable packages. The health of a software project can be affected by the maturity of the packages on which it depends. The version numbers of the used package releases provide an indication of their maturity. Packages with a 0.y.z version number are commonly assumed to be under initial development, suggesting that they are likely to be less stable, and depending on them may be considered as les…
Test automation involves the automatic execution of test scripts instead of being manually run. This significantly reduces the amount of manual effort needed and thus is of great interest to the software testing industry. There are two key problems in the existing tools and methods for test automation - a) Creating an automation test script is essentially a code development task, which most testers are not trained on; and b) the automation test script is seldom readable, making the task of maintenance an effort intensive process. We present the Accelerating Test Automation Platform (ATAP) whic…
In the context of a large software testing company, we have deployed the model-based testing (MBT) approach to take the company's test automation practices to higher levels of maturity /and capability. We have chosen, from a set of open-source/commercial MBT tools, an open-source tool named GraphWalker, and have pragmatically used MBT for end-to-end test automation of several large web and mobile applications under test. The MBT approach has provided, so far in our project, various tangible and intangible benefits in terms of improved test coverage (number of paths tested), improved test-desig…
Code changes constitute one of the most important features of software evolution. Studying them can provide insights into the nature of software development and also lead to practical solutions - recommendations and automations of popular changes for developers. In our work, we developed a tool called PythonChangeMiner that allows to discover code change patterns in the histories of Python projects. We validated the tool and then employed it to discover patterns in the dataset of 120 projects from four different domains of software engineering. We manually categorized patterns that occur in mo…
Continuous practices that rely on automation in the software development workflow have been widely adopted by industry for over a decade. Despite this widespread use, software development remains a primarily human-driven activity that is highly creative and collaborative. There has been extensive research on how continuous practices rely on automation and its impact on software quality and development velocity, but relatively little has been done to understand how automation impacts developer behavior and collaboration. In this paper, we introduce a socio-technical theory about continuous prac…
Background: Meeting the growing industry demand for Data Science requires cross-disciplinary teams that can translate machine learning research into production-ready code. Software engineering teams value adherence to coding standards as an indication of code readability, maintainability, and developer expertise. However, there are no large-scale empirical studies of coding standards focused specifically on Data Science projects. Aims: This study investigates the extent to which Data Science projects follow code standards. In particular, which standards are followed, which are ignored, and how…
For a considerable number of software projects, the creation of effective test cases is hindered by design documentation that is either lacking, incomplete or obsolete. The exploratory testing approach can serve as a sound method in such situations. However, the efficiency of this testing approach strongly depends on the method, the documentation of explored parts of a system, the organization and distribution of work among individual testers on a team, and the minimization of potential (very probable) duplicities in performed tests. In this paper, we present a framework for replacing and auto…
Context: Verification and validation (V&V) activities make up 20 to 50 percent of the total development costs of a software system in practice. Test automation is proposed to lower these V&V costs but available research only provides limited empirical data from industrial practice about the maintenance costs of automated tests and what factors affect these costs. In particular, these costs and factors are unknown for automated GUI-based testing. Objective: This paper addresses this lack of knowledge through analysis of the costs and factors associated with the maintenance of automated GUI-base…
