出其不意策略的回顾与提炼
A Review and Refinement of Surprise Adequacy
摘要：
惊喜验证（SA）是深度学习（DL）测试的新兴和最有前途的充分性标准之一。作为一个充分性准则，它已被用来评估DL测试套件的强度。此外，它还被用来寻找深度神经网络（DNN）的输入，这些输入在训练数据中没有充分表示，或者选择样本进行DNN再训练。然而，测试套件的SA度量的计算可能非常昂贵，因为它涉及到距离计算的二次方。因此，我们开发并发布了一个性能优化但功能等效的SA实现，将评估时间减少了97%。我们还提出了改进的SA计算算法的变体，旨在进一步提高评估速度。然后，我们对MNIST进行了实证研究，重点是SA的分布外检测能力，这使我们能够重现SA首次发布时的部分结果。实验表明，我们改进的变体比普通SA快得多，同时产生类似的结果。我们的实验结果也暴露了SA的一个被忽视的问题：它对与DNN训练过程相关的非确定性非常敏感。
英文摘要：
Surprise Adequacy (SA) is one of the emerging and most promising adequacy criteria for Deep Learning (DL) testing. As an adequacy criterion, it has been used to assess the strength of DL test suites. In addition, it has also been used to find inputs to a Deep Neural Network (DNN) which were not sufficiently represented in the training data, or to select samples for DNN retraining. However, computation of the SA metric for a test suite can be prohibitively expensive, as it involves a quadratic number of distance calculations. Hence, we developed and released a performance-optimized, but functionally equivalent, implementation of SA, reducing the evaluation time by up to 97\%. We also propose refined variants of the SA omputation algorithm, aiming to further increase the evaluation speed. We then performed an empirical study on MNIST, focused on the out-of-distribution detection capabilities of SA, which allowed us to reproduce parts of the results presented when SA was first released. The experiments show that our refined variants are substantially faster than plain SA, while producing comparable outcomes. Our experimental results exposed also an overlooked issue of SA: it can be highly sensitive to the non-determinism associated with the DNN training procedure.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
