The software industry has seen an increasing interest in test automation. In this paper, we present a test automation maturity survey serving as a self-assessment for practitioners. Based on responses of 151 practitioners coming from above 101 organizations in 25 countries, we make observations regarding the state of the practice of test automation maturity: a) The level of test automation maturity in different organizations is differentiated by the practices they adopt; b) Practitioner reported the quite diverse situation with respect to different practices, e.g., 85\% practitioners agreed that their test teams have enough test automation expertise and skills, while 47\% of practitioners admitted that there is lack of guidelines on designing and executing automated tests; c) Some practices are strongly correlated and/or closely clustered; d) The percentage of automated test cases and the use of Agile and/or DevOps development models are good indicators for a higher test automation maturity level; (e) The roles of practitioners may affect response variation, e.g., QA engineers give the most optimistic answers, consultants give the most pessimistic answers. Our results give an insight into present test automation processes and practices and indicate chances for further improvement in the present industry.
The popularity of continuous integration (CI) is increasing as a result of market pressure to release product features or updates frequently. The ability of CI to deliver quality at speed depends on reliable test automation. In this paper, we present an empirical study to observe the effect of test automation maturity (assessed by standard best practices in the literature) on product quality, test automation effort, and release cycle in the CI context of open source projects. We run our test automation maturity survey and got responses from 37 open source java projects. We also mined software …
Test automation is important in software industry but self-assessment instruments for assessing its maturity are not sufficient. The two objectives of this study are to synthesize what an organization should focus to assess its test automation; develop a self-assessment instrument (a survey) for assessing test automation maturity and scientifically evaluate it. We carried out the study in four stages. First, a literature review of 25 sources was conducted. Second, the initial instrument was developed. Third, seven experts from five companies evaluated the initial instrument. Content Validity I…
Mature test automation is key for achieving software quality at speed. In this paper, we present a multivocal literature review with the objective to survey and synthesize the guidelines given in the literature for improving test automation maturity. We selected and reviewed 81 primary studies, consisting of 26 academic literature and 55 grey literature sources. From primary studies, we extracted 26 test automation best practices (e.g., Define an effective test automation strategy, Set up good test environments, Develop high-quality test scripts) and collected many pieces of advice (e.g., in f…
Maturity models for software indicate the key areas that contribute to quality improvements. They usually combine technical, organisational and human aspects relevant for effective software development, to focus the efforts and draw the direction for optimisations. In this paper, we present the process of defining best practices that support the GÉANT Software Maturity Model (GSMM), aligned to the needs of a distributed, innovation-driven, pan-European organisation. Based on the identification of specific goals relevant for GÉANT and a preliminary maturity assessment, we created a catalogue of…
How to successfully conduct test automation process improvement (TAPI) for continuous development, consisting of iterative software development, continuous testing, and delivery, is the challenge faced by many software organizations. In this paper, we present an experience report on TAPI in one DevOps team in F-Secure (a Finnish software company). The team builds Windows application software and exists in F-Secure's TAPI culture. The team self-reports high satisfaction and maturity in test automation for continuous development. To study their TAPI, we reviewed a collection of experience notes,…
We define API Management as an activity that enables organizations to design, publish and deploy their APIs for (external) developers to consume. API Management capabilities such as controlling API lifecycles, access and authentication to APIs, monitoring, throttling and analyzing API usage, as well as providing security and documentation. These capabilities are often implemented through an integrated platform. This data set describes the API Management Focus Area Maturity Model (API-m-FAMM). In a structured manner, this model aims to support organizations that expose their API(s) to third-par…
With the agile approach to managing software development projects comes an increased dependability on well functioning teams, since many of the practices are built on teamwork. The objective of this study was to investigate if, and how, team development from a group psychological perspective is related to some work practices of agile teams. Data were collected from 34 agile teams (200 individuals) from six software development organizations and one university in both Brazil and Sweden using the Group Development Questionnaire (Scale IV) and the Perceptive Agile Measurement (PAM). The result in…
Game development is an interdisciplinary concept that embraces artistic, software engineering, management, and business disciplines. This research facilitates a better understanding of important dimensions of digital game development methodology. Game development is considered as one of the most complex tasks in software engineering. The increased popularity of digital games, the challenges faced by game development organizations in developing quality games, and high competition in the digital game industry demand a game development maturity assessment. Consequently, this study presents a Digi…
Context: Existing knowledge in agile software development suggests that individual competency (e.g. skills) is a critical success factor for agile projects. While assuming that technical skills are important for every kind of software development project, many researchers suggest that non-technical individual skills are especially important in agile software development. Objective: In this paper, we investigate whether non-technical individual skills can predict the use of agile practices. Method: Through creating a set of multiple linear regression models using a total of 113 participants fro…
The fundamental objective of this research work is to develop a Software Requirement Engineering Healthcare Implementation Maturity Model (SRE-HIMM) that will assist healthcare organizations to effectively evaluate and implement HIS development process. The model was developed based on the systematic literature review (SLR) approach and Empirical results. The 53 primary studies were extracted using the SLR approach and CSFs, CBs, and best practices were identified form the extracted primary studies. The identified success factors and barriers were further ranked using the analytical hierarchy …
The focus on psychology has increased within software engineering due to the project management innovation "agile development processes". The agile methods do not explicitly consider group development aspects; they simply assume what is described in group psychology as mature groups. This study was conducted with 45 employees and their twelve managers (N=57) from two SAP customers in the US that were working with agile methods, and the data were collected via an online survey. The selected Agility measurement was correlated to a Group Development measurement and showed significant convergent v…
The integration of industrial automation systems and software agents has been practiced for many years. However, such an integration is usually done by experts and there is no consistent way to assess these practices and to optimally select one for a specific system. Standards such as the ISO/IEC 25023 propose measures that could be used to obtain a quantification on the characteristics of such integration. In this work, the suitability of these characteristics and their proposed calculation for assessing the connection of industrial automation systems with software agents is discussed. Result…
Context: User-Centered Design and Agile methodologies focus on human issues. Nevertheless, agile methodologies focus on contact with contracting customers and generating value for them. Usually, the communication between end users and the agile team is mediated by customers. However, they do not know the problems end users face in their routines. Hence, UX issues are typically identified only after the implementation, during user testing and validation. Objective: Aiming to improve the understanding and definition of the problem in agile projects, this research investigates the practices and d…
Continuous practices that rely on automation in the software development workflow have been widely adopted by industry for over a decade. Despite this widespread use, software development remains a primarily human-driven activity that is highly creative and collaborative. There has been extensive research on how continuous practices rely on automation and its impact on software quality and development velocity, but relatively little has been done to understand how automation impacts developer behavior and collaboration. In this paper, we introduce a socio-technical theory about continuous prac…
Several studies have shown that automated support for different activities of the security patch management process has great potential for reducing delays in installing security patches. However, it is also important to understand how automation is used in practice, its limitations in meeting real-world needs and what practitioners really need, an area that has not been empirically investigated in the existing software engineering literature. This paper reports an empirical study aimed at investigating different aspects of automation for security patch management using semi-structured intervi…
Software process improvement (SPI) is a means to an end, not an end in itself (e.g., a goal is to achieve shorter time to market and not just compliance to a process standard). Therefore, SPI initiatives ought to be streamlined to meet the desired values for an organization. Through a literature review, seven secondary studies aggregating maturity models and assessment frameworks were identified. Furthermore, we identified six proposals for building a new maturity model. We analyzed the existing maturity models for (a) their purpose, structure, guidelines, and (b) the degree to which they expl…
Continuous practices are a staple of the modern software development workflow. Automation, in particular, is widely adopted due to its benefits related to quality and productivity. However, automation, similarly to all other aspects of the software development workflow, interacts with humans (in this case developers). While some work has investigated the impact of automation on developers, it is not clear to what extent context and process influence that impact. We present our ADEPT theory of developers and automation, in an attempt to bridge this gap and identify the possible ways context, pr…
This chapter surveys the state of the art in code coverage from the perspective of test automation. Our aim is to describe and motivate the three most popular classes of glass box test coverage models, which are: control flow, logic and data flow coverage. We take a fairly rigorous approach to code coverage models. Thus, for each class, we will give precise definitions of specific examples, some of which are widely known while others deserve to be better known by test engineers. Our main goal is to present coverage models that represent the state of the art. These should stimulate thought rega…
Agent-technologies have been used for higher-level decision making in addition to carrying out lower-level automation and control functions in industrial systems. Recent research has identified a number of architectural patterns for the use of agents in industrial automation systems but these practices vary in several ways, including how closely agents are coupled with physical systems and their control functions. Such practices may play a pivotal role in the Cyber-Physical System integration and interaction. Hence, there is a clear need for a common set of criteria for assessing available pra…
Maintenance of Automated Test Suites in Industry: An Empirical study on Visual GUI Testing
验证与确认（Verification and Validation，V&V）活动在实践中占软件系统总开发成本的20%到50%。测试自动化被提出来降低这些V&V成本，但现有的研究只提供了有限的经验数据，从工业实践中的维护成本的自动化测试和影响这些成本的因素。特别是，这些成本和因素对于基于GUI的自动化测试是未知的。目的：本文通过分析与工业实践中基于GUI的自动化测试维护相关的成本和因素，解决了这种知识的缺乏。方法：在两个公司，西门子和萨博，实证研究报告的采访，和实证工作，可视化GUI测试进行获取数据的技术的维护成本和可行性。结果：观察到13个因素，影响维修，如测试人员的知识/经验和测试用例的复杂性。此外，统计分析表明，开发新的测试脚本比维护成本更高，但频繁的维护比不频繁的大爆炸维护成本更低。此外，成本模型，在以前的工作的基础上，提出了估计的时间，以积极的投资回报率（ROI）的测试自动化相比，手动测试。结论：它的结论是，测试自动化可以降低一个项目的整体软件开发成本，同时也对软件质量有积极的影响。然而，维护成本仍然是相当可观的，公司目前花在手动测试上的时间越少，在自动化之后达到积极的、经济的ROI之前需要的时间就越多。
Test automation involves the automatic execution of test scripts instead of being manually run. This significantly reduces the amount of manual effort needed and thus is of great interest to the software testing industry. There are two key problems in the existing tools and methods for test automation - a) Creating an automation test script is essentially a code development task, which most testers are not trained on; and b) the automation test script is seldom readable, making the task of maintenance an effort intensive process. We present the Accelerating Test Automation Platform (ATAP) whic…
Microservice architecture advocates a number of technologies and practices such as lightweight container, container orchestration, and DevOps, with the promised benefits of faster delivery, improved scalability, and greater autonomy. However, microservice systems implemented in industry vary a lot in terms of adopted practices and achieved benefits, drastically different from what is advocated in the literature. In this article, we conduct an empirical study, including an online survey with 51 responses and 14 interviews for experienced microservice experts to advance our understanding regardi…
In the context of a large software testing company, we have deployed the model-based testing (MBT) approach to take the company's test automation practices to higher levels of maturity /and capability. We have chosen, from a set of open-source/commercial MBT tools, an open-source tool named GraphWalker, and have pragmatically used MBT for end-to-end test automation of several large web and mobile applications under test. The MBT approach has provided, so far in our project, various tangible and intangible benefits in terms of improved test coverage (number of paths tested), improved test-desig…
This paper sets the context for the urgency for cyber autonomy, and the current gaps of the cyber security industry. A novel framework proposing four phases of maturity for full cyber autonomy will be discussed. The paper also reviews new and emerging cyber security automation techniques and tools, and discusses their impact on society, the perceived cyber security skills gap/shortage and national security. We will also be discussing the delicate balance between national security, human rights and ethics, and the potential demise of the manual penetration testing industry in the face of automa…
Overcoming Challenges in Agile and DevOps Integration: A Qualitative Study
University of Applied Sciences Emden/Leer（埃姆登/莱尔应用科学大学）；Auckland University of Technology（奥克兰理工大学）；University of Applied Sciences & Arts Hannover（汉诺威应用科学与艺术大学）
Marketplaces for distributing software products and services have been getting increasing popularity. GitHub, which is most known for its version control functionality through Git, launched its own marketplace in 2017. GitHub Marketplace hosts third party apps and actions to automate workflows in software teams. Currently, this marketplace hosts 440 Apps and 7,878 Actions across 32 different categories. Overall, 419 Third party developers released their apps on this platform which 111 distinct customers adopted. The popularity and accessibility of GitHub projects have made this platform and th…
Context: New software development patterns are emerging aiming at accelerating the process of delivering value. One is Continuous Experimentation, which allows to systematically deploy and run instrumented software variants during development phase in order to collect data from the field of application. While currently this practice is used on a daily basis on web-based systems, technical difficulties challenge its adoption in fields where computational resources are constrained, e.g., cyber-physical systems and the automotive industry. Objective: This paper aims at providing an overview of th…
DUCTILE: Agentic LLM Orchestration of Engineering Analysis in Product Development Practice
Product Development Division, Department of Mechanical Engineering, Chalmers University of Technology, Gothenburg, Sweden（产品开发部门，机械工程系，楚姆勒斯技术大学，哥德堡，瑞典）；GKN Aerospace Engine Systems, Trollhättan, Sweden（GKN航空发动机系统，特罗尔哈塔纳，瑞典）
Background. DevOps is a set of practices and cultural values that aims to reduce the barriers between development and operations teams. Due to its increasing interest and imprecise definitions, existing research works have tried to characterize DevOps---mainly using a set of concepts and related practices. Aims. Nevertheless, little is known about thepractitioners practitioners' understanding about successful paths for DevOps adoption. The lack of such understanding might hinder institutions to adopt DevOps practices. Therefore, our goal here is to present a theory about DevOps adoption, highl…
The ongoing fourth Industrial Revolution depends mainly on robust Industrial Cyber-Physical Systems (ICPS). ICPS includes computing (software and hardware) abilities to control complex physical processes in distributed industrial environments. Industrial agents, originating from the well-established multi-agent systems field, provide complex and cooperative control mechanisms at the software level, allowing us to develop larger and more feature-rich ICPS. The IEEE P2660.1 standardisation project, "Recommended Practices on Industrial Agents: Integration of Software Agents and Low Level Automati…
Adopting DevOps practices is nowadays a recurring task in the industry. DevOps is a set of practices intended to reduce the friction between the software development (Dev) and the IT operations (Ops), resulting in higher quality software and a shorter development lifecycle. Even though many resources are talking about DevOps practices, they are often inconsistent with each other on the best DevOps practices. Furthermore, they lack the needed detail and structure for beginners to the DevOps field to quickly understand them. In order to tackle this issue, this paper proposes four foundational De…
GitHub introduced "Actions" in 2019 to increase workflow velocity and add customized automation to the repositories. Any individual can develop Actions for automating workflow on GitHub repositories and others can reuse them whenever open source. GitHub introduced its marketplace for commercializing and sharing these automation tools, which currently hosts 16,730 Actions. Yet, there are numerous Actions that are developed and distributed in local repositories and outside the Marketplace. So far, the research community conducted mining studies to understand Actions with a significant focus on C…
