测量区分度以促进多个深度学习模型的比较测试
Measuring Discrimination to Boost Comparative Testing for Multiple Deep Learning Models
摘要：
DL技术的兴起导致了大量DL模型的建立和共享，这为DL模型的获取和重用提供了便利。对于一个给定的任务，我们会遇到具有相同功能的多个DL模型，这些模型被认为是实现该任务的候选模型。预计测试人员将比较多个DL模型，并选择更合适的w.r.t.整个测试环境。由于标签工作的限制，测试人员的目标是选择一个有效的样本子集，以尽可能精确地估计这些模型的秩。为了解决这个问题，我们提出了基于样本区分的选择（SDS）来选择可以区分多个模型的有效样本，即，这些样本的预测行为（正确/错误）将有助于指示模型性能的趋势。为了评估SDS，我们使用三个广泛使用的图像数据集和80个真实世界的DL模型进行了广泛的实证研究。实验结果表明，与现有的基线方法相比，SDS是一种有效和高效的样本选择方法来排序多个DL模型。
英文摘要：
The boom of DL technology leads to massive DL models built and shared, which facilitates the acquisition and reuse of DL models. For a given task, we encounter multiple DL models available with the same functionality, which are considered as candidates to achieve this task. Testers are expected to compare multiple DL models and select the more suitable ones w.r.t. the whole testing context. Due to the limitation of labeling effort, testers aim to select an efficient subset of samples to make an as precise rank estimation as possible for these models. To tackle this problem, we propose Sample Discrimination based Selection (SDS) to select efficient samples that could discriminate multiple models, i.e., the prediction behaviors (right/wrong) of these samples would be helpful to indicate the trend of model performance. To evaluate SDS, we conduct an extensive empirical study with three widely-used image datasets and 80 real world DL models. The experimental results show that, compared with state-of-the-art baseline methods, SDS is an effective and efficient sample selection method to rank multiple DL models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
