Flakify：一个基于语言模型的黑盒预测器
Flakify: A Black-Box, Language Model-based Predictor for Flaky Tests
摘要：
软件测试确保代码变更不会对现有功能产生不利影响。然而，测试用例可以是可扩展的，即，在执行过程中传递和失败，即使是同一版本的源代码。不稳定的测试用例会给软件开发带来额外的开销，因为它们会导致不必要的调试生产或测试代码的尝试。最先进的基于ML的测试用例预测器依赖于预定义的功能集，这些功能集要么是特定于项目的，要么需要访问生产代码，而软件测试工程师并不总是可以访问生产代码。因此，在本文中，我们提出了Flakify，一个黑盒，语言模型为基础的预测器，用于测试用例。Flakify完全依赖于测试用例的源代码，因此不需要（a）访问生产代码（黑盒），（b）检查测试用例，（c）预定义功能。为此，我们采用了CodeBERT，一个预先训练好的语言模型，并对其进行了微调，以使用测试用例的源代码来预测JavaScript测试用例。我们在两个公开的数据集（FlakeFlagger和IDoFT）上评估了Flakify的测试用例，并使用两种不同的评估程序（交叉验证和按项目验证）将我们的技术与FlakeFlagger方法进行了比较。Flakify在使用交叉验证和按项目验证的两个数据集上都获得了很高的F1分数，并且在FlakeFlagger数据集上进行评估时，在精度和召回率方面分别超过了FlakeFlagger 10和18个百分点，从而减少了不必要的调试测试用例和生产代码的成本。当用于预测新项目的测试用例时，Flakify也取得了显着更高的预测结果，这表明FlakeFlagger具有更好的泛化能力。我们的研究结果进一步表明，黑盒版本的FlakeFlagger是不是一个可行的选择，用于预测测试用例。
英文摘要：
Software testing assures that code changes do not adversely affect existing functionality. However, a test case can be flaky, i.e., passing and failing across executions, even for the same version of the source code. Flaky test cases introduce overhead to software development as they can lead to unnecessary attempts to debug production or testing code. The state-of-the-art ML-based flaky test case predictors rely on pre-defined sets of features that are either project-specific, require access to production code, which is not always available to software test engineers. Therefore, in this paper, we propose Flakify, a black-box, language model-based predictor for flaky test cases. Flakify relies exclusively on the source code of test cases, thus not requiring to (a) access to production code (black-box), (b) rerun test cases, (c) pre-define features. To this end, we employed CodeBERT, a pre-trained language model, and fine-tuned it to predict flaky test cases using the source code of test cases. We evaluated Flakify on two publicly available datasets (FlakeFlagger and IDoFT) for flaky test cases and compared our technique with the FlakeFlagger approach using two different evaluation procedures: cross-validation and per-project validation. Flakify achieved high F1-scores on both datasets using cross-validation and per-project validation, and surpassed FlakeFlagger by 10 and 18 percentage points in terms of precision and recall, respectively, when evaluated on the FlakeFlagger dataset, thus reducing the cost bound to be wasted on unnecessarily debugging test cases and production code by the same percentages. Flakify also achieved significantly higher prediction results when used to predict test cases on new projects, suggesting better generalizability over FlakeFlagger. Our results further show that a black-box version of FlakeFlagger is not a viable option for predicting flaky test cases.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
