从错误警报中辨别合理故障：Chromium的持续集成研究
Discerning Legitimate Failures From False Alerts: A Study of Chromium's Continuous Integration
摘要：
在软件测试中，片状是一个主要的问题。对于同一个程序版本，不可靠的测试通过和失败，误导了那些花费时间和资源调查测试失败的开发人员，结果发现它们是错误的警报。在实践中，解决这个问题的实际方法是检查失败的测试，希望它们能够通过并表现为错误警报。尽管如此，完全过滤掉错误警报可能需要不成比例的重新运行次数，因此在计算和时间方面都产生了重要的成本。作为一种替代重播，我们提出公平，一种新颖的，轻量级的方法，分类测试失败到假警报和合法的失败。Fair依赖于一个分类器和一组来自故障和测试伪影的特征。为了构建和评估我们的机器学习分类器，我们使用了Chromium项目的持续集成。特别是，我们从2,000个构建中收集了超过100万个测试失败的属性和伪影。我们的研究结果表明，公平可以准确地区分合法的故障从虚假警报，与MCC高达95%。此外，通过研究不同的测试类别：图形用户界面，集成和单元测试，我们表明，公平准确地分类故障，即使故障的数量是有限的。最后，我们比较了我们的方法重新运行的成本，并表明公平可以节省高达20分钟的计算时间，每次构建。
英文摘要：
Flakiness is a major concern in Software testing. Flaky tests pass and fail for the same version of a program and mislead developers who spend time and resources investigating test failures only to discover that they are false alerts. In practice, the defacto approach to address this concern is to rerun failing tests hoping that they would pass and manifest as false alerts. Nonetheless, completely filtering out false alerts may require a disproportionate number of reruns, and thus incurs important costs both computation and time-wise. As an alternative to reruns, we propose Fair, a novel, lightweight approach that classifies test failures into false alerts and legitimate failures. Fair relies on a classifier and a set of features from the failures and test artefacts. To build and evaluate our machine learning classifier, we use the continuous integration of the Chromium project. In particular, we collect the properties and artefacts of more than 1 million test failures from 2,000 builds. Our results show that Fair can accurately distinguish legitimate failures from false alerts, with an MCC up to 95%. Moreover, by studying different test categories: GUI, integration and unit tests, we show that Fair classifies failures accurately even when the number of failures is limited. Finally, we compare the costs of our approach to reruns and show that Fair could save up to 20 minutes of computation time per build.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
