基于通用代码演化和测试历史数据的实际片状测试预测
Practical Flaky Test Prediction using Common Code Evolution and Test History Data
摘要：
非确定性行为的测试用例会导致开发人员对他们的回归测试套件失去信任，并最终忽略失败。因此，检测漏洞测试是维护代码质量的一项关键任务，因为它为任何形式的系统响应建立了必要的基础，例如测试验证或自动调试。以前的研究已经提出了各种方法来检测flakiness，但当试图在工业环境中部署这些方法时，它们对仪器，测试重新运行或特定于语言的工件的依赖是令人望而却步的。因此，在本文中，我们调查的预测，没有这样的要求，底层的程序设计语言，CI，构建或测试执行框架的测试。相反，我们只依赖于最常用的工件，即测试的结果和持续时间，以及关于代码演变的基本信息，来构建能够检测脆弱性的预测模型。此外，我们的方法不需要额外的重新运行，因为它从现有的测试执行中收集这些数据。我们根据建议的功能训练了几个已建立的分类器，并在一个大型工业软件系统上评估了它们的性能，从中我们收集了100个测试和100个非测试和代码历史的数据集。最好的模型仅使用3个特征就能够达到95.5%的F1分数：测试的翻转率，过去54天内对源文件的更改数量，以及最近一次拉取请求中更改的文件数量。
英文摘要：
Non-deterministically behaving test cases cause developers to lose trust in their regression test suites and to eventually ignore failures. Detecting flaky tests is therefore a crucial task in maintaining code quality, as it builds the necessary foundation for any form of systematic response to flakiness, such as test quarantining or automated debugging. Previous research has proposed various methods to detect flakiness, but when trying to deploy these in an industrial context, their reliance on instrumentation, test reruns, or language-specific artifacts was inhibitive. In this paper, we therefore investigate the prediction of flaky tests without such requirements on the underlying programming language, CI, build or test execution framework. Instead, we rely only on the most commonly available artifacts, namely the tests' outcomes and durations, as well as basic information about the code evolution to build predictive models capable of detecting flakiness. Furthermore, our approach does not require additional reruns, since it gathers this data from existing test executions. We trained several established classifiers on the suggested features and evaluated their performance on a large-scale industrial software system, from which we collected a data set of 100 flaky and 100 non-flaky test- and code-histories. The best model was able to achieve an F1-score of 95.5% using only 3 features: the tests' flip rates, the number of changes to source files in the last 54 days, as well as the number of changed files in the most recent pull request.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
