SAP HANA背景下易失败测试的词汇
The Vocabulary of Flaky Tests in the Context of SAP HANA
AI总结：
本研究在SAP HANA背景下评估了基于源代码标识符识别易失败测试的方法，通过复现和改进现有技术，提高了识别准确率，但实际应用中仍受限。
摘要：
背景。自动化测试执行是收集软件项目质量信息的重要活动。然而，所谓的易失败测试却会负面影响这一过程。此类测试似乎随机失败，而代码未发生变化，因此无法提供清晰的信号。先前的研究提出基于测试代码中的源代码标识符来识别易失败测试。迄今为止，这些方法尚未在大规模工业环境中进行评估。目标。我们评估了基于测试代码中的源代码标识符来识别易失败测试及其根本原因的方法，在大规模工业项目中。方法。首先，我们在SAP HANA背景下复现了Pinto等人之前的工作。其次，我们评估了不同的特征提取技术，即TF-IDF和TF-IDFC-RF。第三，我们评估了CodeBERT和XGBoost作为分类模型。为了保证比较的严谨性，我们利用了先前工作的数据集以及两个SAP HANA的数据集。结果。我们的复现显示在原始数据集和SAP HANA的一个数据集上结果相似。虽然原始方法在原始数据集上获得了F1-Score为0.94，在SAP HANA数据集上为0.92，我们的扩展方法分别达到了0.96和0.99的F1-Score。在SAP HANA背景下，依赖外部数据源是导致测试易失败的常见根本原因。结论。大规模工业项目中的词汇似乎在精确术语上略有不同，但术语的类别，如远程依赖，与之前的实证发现相似。然而，即使F1-Score较大，找到易失败的源代码标识符和黑盒预测在实践中也有有限的用途，因为结果对开发者来说并不具有可操作性。
英文摘要：
Background. Automated test execution is an important activity to gather information about the quality of a software project. So-called flaky tests, however, negatively affect this process. Such tests fail seemingly at random without changes to the code and thus do not provide a clear signal. Previous work proposed to identify flaky tests based on the source code identifiers in the test code. So far, these approaches have not been evaluated in a large-scale industrial setting. Aims. We evaluate approaches to identify flaky tests and their root causes based on source code identifiers in the test code in a large-scale industrial project. Method. First, we replicate previous work by Pinto et al. in the context of SAP HANA. Second, we assess different feature extraction techniques, namely TF-IDF and TF-IDFC-RF. Third, we evaluate CodeBERT and XGBoost as classification models. For a sound comparison, we utilize both the data set from previous work and two data sets from SAP HANA. Results. Our replication shows similar results on the original data set and on one of the SAP HANA data sets. While the original approach yielded an F1-Score of 0.94 on the original data set and 0.92 on the SAP HANA data set, our extensions achieve F1-Scores of 0.96 and 0.99, respectively. The reliance on external data sources is a common root cause for test flakiness in the context of SAP HANA. Conclusions. The vocabulary of a large industrial project seems to be slightly different with respect to the exact terms, but the categories for the terms, such as remote dependencies, are similar to previous empirical findings. However, even with rather large F1-Scores, both finding source code identifiers for flakiness and a black box prediction have limited use in practice as the results are not actionable for developers.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
