基于少样本学习的Flaky测试类别预测
Predicting Flaky Tests Categories using Few-Shot Learning
摘要：
Flaky测试是在同一版本的程序上运行时产生不同结果的测试。这种不确定性行为会给持续集成带来错误信号，浪费开发人员的时间，降低他们对测试套件的信任。研究强调了保持测试无剥落的重要性。近年来，研究界提出了许多静态和动态的方法来推进对伪测试的检测。虽然有希望，但这些方法主要集中在将测试分类为合格或不合格，即使报告了高性能，理解片状的原因仍然具有挑战性。这一部分对于旨在解决它的研究人员和开发人员至关重要。为了帮助理解给定的故障测试，我们提出了FlakyCat，这是第一种基于根本原因类别对故障测试进行分类的方法。FlakyCat依赖于CodeBERT进行代码表示，并利用基于Siamese网络的少样本学习方法来训练具有少量数据的多类分类器。我们对FlakyCat进行了训练，并在一组从开源Java项目中收集的343项测试中对FlakyCat进行了评估。我们的评估表明，FlakyCat准确地对重复测试进行分类，加权F1得分为70%。此外，我们研究了我们的方法对每个类别的性能，揭示了异步等待，无序收集和时间相关的测试是准确分类，而并发相关的测试是更具有挑战性的预测。最后，为了便于理解FlakyCat的预测，我们提出了一种新的技术CodeBERT为基础的模型的可解释性，突出代码语句影响的分类。
英文摘要：
Flaky tests are tests that yield different outcomes when run on the same version of a program. This non-deterministic behaviour plagues continuous integration with false signals, wasting developers' time and reducing their trust in test suites. Studies highlighted the importance of keeping tests flakiness-free. Recently, the research community has been pushing forward the detection of flaky tests by suggesting many static and dynamic approaches. While promising, those approaches mainly focus on classifying tests as flaky or not and, even when high performances are reported, it remains challenging to understand the cause of flakiness. This part is crucial for researchers and developers that aim to fix it. To help with the comprehension of a given flaky test, we propose FlakyCat, the first approach for classifying flaky tests based on their root cause category. FlakyCat relies on CodeBERT for code representation and leverages a Siamese network-based Few-Shot learning method to train a multi-class classifier with few data. We train and evaluate FlakyCat on a set of 343 flaky tests collected from open-source Java projects. Our evaluation shows that FlakyCat categorises flaky tests accurately, with a weighted F1 score of 70%. Furthermore, we investigate the performance of our approach for each category, revealing that Async waits, Unordered collections and Time-related flaky tests are accurately classified, while Concurrency-related flaky tests are more challenging to predict. Finally, to facilitate the comprehension of FlakyCat's predictions, we present a new technique for CodeBERT-based model interpretability that highlights code statements influencing the categorization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
