完美是测试预言机的敌人
Perfect is the enemy of test oracle
AI总结:
本文提出基于学习的测试预言机方法SEER,通过联合嵌入单元测试与测试方法构建分类器,在无需确切正确行为的情况下准确预测测试通过或失败,并在5000多个Java单元测试上验证了其高效性、准确性与泛化能力。
摘要:
测试预言机的自动化是软件测试中最具挑战性的方面之一,但与自动化测试输入生成相比,受到的关注相对较少。测试预言机依赖于一个能够区分正确和错误行为的真值,以确定测试是失败(检测到缺陷)还是通过。使得预言机问题具有挑战性且不可判定的原因在于,它假设真值应该知道确切的预期、正确或错误的行为。然而,我们认为,即使不知道确切的正确或错误行为,只要知道这两种行为可能有何不同,仍然可以构建准确的预言机。本文提出了SEER,一种基于学习的方法,在缺少测试断言或其他类型预言机的情况下,能够确定单元测试在给定测试方法(MUT)上是通过还是失败。为了构建真值,SEER将单元测试和MUT的实现联合嵌入到一个统一的向量空间中,使得通过的测试的神经表示与其对应的MUT相似,而失败的测试的神经表示与其对应的MUT不相似。在此向量表示之上构建的分类器作为预言机,当测试输入检测到MUT中的缺陷时生成“失败”标签,否则生成“通过”标签。我们将SEER应用于来自不同开源Java项目的5000多个单元测试的大量实验表明,生成的预言机在预测失败或通过标签方面是有效的,总体准确率、精确率、召回率和F1分数分别为93%、86%、94%和90%;具有泛化能力,对不在训练或验证集中的项目的单元测试进行标签预测时,性能下降微乎其微;并且高效,平均仅需6.5毫秒即可检测出缺陷的存在。
英文摘要:
Automation of test oracles is one of the most challenging facets of software testing, but remains comparatively less addressed compared to automated test input generation. Test oracles rely on a ground-truth that can distinguish between the correct and buggy behavior to determine whether a test fails (detects a bug) or passes. What makes the oracle problem challenging and undecidable is the assumption that the ground-truth should know the exact expected, correct, or buggy behavior. However, we argue that one can still build an accurate oracle without knowing the exact correct or buggy behavior, but how these two might differ. This paper presents SEER, a learning-based approach that in the absence of test assertions or other types of oracle, can determine whether a unit test passes or fails on a given method under test (MUT). To build the ground-truth, SEER jointly embeds unit tests and the implementation of MUTs into a unified vector space, in such a way that the neural representation of tests are similar to that of MUTs they pass on them, but dissimilar to MUTs they fail on them. The classifier built on top of this vector representation serves as the oracle to generate "fail" labels, when test inputs detect a bug in MUT or "pass" labels, otherwise. Our extensive experiments on applying SEER to more than 5K unit tests from a diverse set of open-source Java projects show that the produced oracle is (1) effective in predicting the fail or pass labels, achieving an overall accuracy, precision, recall, and F1 measure of 93%, 86%, 94%, and 90%, (2) generalizable, predicting the labels for the unit test of projects that were not in training or validation set with negligible performance drop, and (3) efficient, detecting the existence of bugs in only 6.5 milliseconds on average.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。