There are many widely used tools for measuring test-coverage and code-coverage. Test coverage is the ratio of requirements or other non-code artifacts covered by a test suite, while code-coverage is the ratio of source code covered by tests. Almost all coverage tools show a few certain subset of coverage values, and almost always either test-coverage or code-coverage measures. In a large-scale industrial web-application-testing setting, we were faced with the need to "integrate" several types of coverage data (including front-end and back-end code coverage with requirements coverage), and to see all of them "live" as large model-based test suites were running. By being unable to find any off-the-shelf toolset to address the above need, we have developed an open-source test coverage tool, specific for MBT, named MBTCover. In addition to code coverage, the tool measures and reports requirements and model coverage, "live" as a given MBT test suite is executing. In this paper, we present the features of the MBTCover tool and our experience from using it in multiple large test-automation projects in practice. Other software test engineers, who conduct web application testing and MBT, may find the tool useful in their projects.
The state-of-the-practice in software development is driven by constant change fueled by continuous integration servers. Such constant change demands for frequent and fully automated tests capable to detect faults immediately upon project build. As the fault detection capability of the test suite becomes so important, modern software development teams continuously monitor the quality of the test suite as well. However, it appears that the state-of-the-practice is reluctant to adopt strong coverage metrics (namely mutation coverage), instead relying on weaker kinds of coverage (namely branch co…
An approach is introduced, which supports a testing technician in the identification of possibly untested behavior of control software of fully integrated automated production systems (aPS). Based on an approach for guided semi-automatic system testing, execution traces are recorded during testing, allowing a subsequent coverage assessment. As the behavior of an aPS is highly dependent on the software, omitted system behavior can be identified and assessed for criticality. Through close cooperation with industry, this approach represents the first coverage assessment approach for system testin…
ResTest: Targeted Coverage of Residual Not-Covered Code Using Large Language Models for Web GUI Testing
Key Lab of HCST (PKU), MOE（北京大学）；SCS（北京大学）；Peking University, Beijing, China（中国科学院软件研究所；中国科学院大学）；Key Lab of HCST (PKU) & NERC of SE, MOE（北京通明湖信息技术应用创新中心）；Peking University, Beijing, China（复旦大学高级计算系统研究所）；Institute of Software Chinese Academy of Sciences, University of Chinese Academy of Sciences, Beijing, China；Beijing Tongming Lake Information Technology Application Innovation Center (TLAIC), China；Fudan University Institute of Systems for Advanced Computing, China
Automatic testing of mobile applications has been a well-researched area in recent years. However, testing in industry is still a very manual practice, as research results have not been fully transferred and adopted. Considering mobile applications, manual testing has the additional burden of adequate testing posed by a large number of available devices and different configurations, as well as the maintenance and setup of such devices. In this paper, we propose and evaluate the use of a model-based test generation approach, where generated tests are executed on a set of cloud-hosted real mobil…
Understanding Automated Web GUI Testing: An Empirical Study Across Exploration Strategies and State Abstractions
SCS, Peking University（北京大学计算科学研究所）；Key Lab of HCST (PKU), MOE（北京大学智能技术与系统国家重点实验室）；University of Texas at Dallas（德克萨斯大学达拉斯分校）；National Engineering Research Center of Software Engineering, MOE（软件工程国家工程研究中心）
In the context of a large software testing company, we have deployed the model-based testing (MBT) approach to take the company's test automation practices to higher levels of maturity /and capability. We have chosen, from a set of open-source/commercial MBT tools, an open-source tool named GraphWalker, and have pragmatically used MBT for end-to-end test automation of several large web and mobile applications under test. The MBT approach has provided, so far in our project, various tangible and intangible benefits in terms of improved test coverage (number of paths tested), improved test-desig…
Mutation testing is used to evaluate the effectiveness of test suites. In recent years, a promising variation called extreme mutation testing emerged that is computationally less expensive. It identifies methods where their functionality can be entirely removed, and the test suite would not notice it, despite having coverage. These methods are called pseudo-tested. In this paper, we compare the execution and analysis times for traditional and extreme mutation testing and discuss what they mean in practice. We look at how extreme mutation testing impacts current software development practices a…
Technical University of Munich（慕尼黑工业大学）；Università degli Studi di Napoli Federico II（那不勒斯费德里科二世大学）；University of St. Gallen（圣加仑大学）；Università della Svizzera italiana（瑞士意大利语区大学）
TraceLib: System-Call Bitmap Feedback Mechanism for Language-Agnostic Web Fuzzing
Bernoulli Institute, University of Groningen（格罗宁根大学伯努利学院）；Faculty of Engineering, Udayana University（乌达亚纳大学工程学院）；Department of Computer Science, University of Cyprus（塞浦路斯大学计算机科学系）
Remote Procedure Call (RPC) is a communication protocol to support client-server interactions among services over a network. RPC is widely applied in industry for building large-scale distributed systems, such as Microservices. Modern RPC frameworks include for example Thrift, gRPC, SOFARPC and Dubbo. Testing such systems is very challenging, due to the complexity of distributed systems and various RPC frameworks the system could employ. To the best of our knowledge, there does not exist any tool or solution that could enable automated testing of modern RPC-based services. To fill this gap, in…
Web testing has long been recognized as a notoriously difficult task. Even nowadays, web testing still heavily relies on manual efforts while automated web testing is far from achieving human-level performance. Key challenges in web testing include dynamic content update and deep bugs hiding under complicated user interactions and specific input values, which can only be triggered by certain action sequences in the huge search space. In this paper, we propose WebExplor, an automatic end-to-end web testing framework, to achieve an adaptive exploration of web applications. WebExplor adopts curio…
We present a novel idea on adequacy testing called ``{vulnerability coverage}.'' The introduced coverage measure examines the underlying software for the presence of certain classes of vulnerabilities often found in the National Vulnerability Database (NVD) website. The thoroughness of the test input generation procedure is performed through the adaptation of evolutionary algorithms namely Genetic Algorithms (GA) and Particle Swarm Optimization (PSO). The methodology utilizes the Common Vulnerability Scoring System (CVSS), a free and open industry standard for assessing the severity of compute…
University of Bristol（布里斯托大学）；Infineon Technologies（英飞凌科技）
Constrained random test generation is one of the most widely adopted methods for generating stimuli for simulation-based verification. Randomness leads to test diversity, but tests tend to repeatedly exercise the same design logic. Constraints are written (typically manually) to bias random tests towards interesting, hard-to-reach, and yet-untested logic. However, as verification progresses, most constrained random tests yield little to no effect on functional coverage. If stimuli generation consumes significantly less resources than simulation, then a better approach involves randomly generat…
