arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2007.01472cs.LGstat.ML
收藏

通过精度监控提高深度神经网络的可信度

Increasing Trustworthiness of Deep Neural Networks via Accuracy Monitoring

University of California, Riverside(加州大学河滨分校)

Zhihui Shao, Jianyi Yang, Shaolei Ren

摘要:

深度神经网络(DNN)的推理准确性是一个至关重要的性能指标,但在实践中可能会因实际测试数据集而发生很大变化,并且由于缺乏地面真值标签而通常是未知的。这引起了人们对DNN可信度的严重关注,特别是在安全关键型应用中。在本文中,我们通过使用事后处理来监控用户数据集上的真实推理准确性来解决DNN的可信度。具体地说,我们提出了一个基于神经网络的准确性监测模型,它只需要部署DNN的softmax概率输出作为其输入,并直接预测DNN的预测结果是否正确,从而导致对真实推理准确性的估计。准确度监控模型可以在与感兴趣的目标应用相关的数据集上进行预训练,并且只需要主动标记用户数据集的一小部分(在我们的实验中为1%)用于模型传输。对于估计的鲁棒性,我们进一步采用基于Monte-Carlo dropout方法的监测模型的集合。我们在不同部署的DNN模型上评估了我们的方法,用于多个数据集(包括对抗样本)上的图像分类和交通标志检测。结果表明,我们的精度监控模型提供了一个接近真实的精度估计,并优于现有的基线方法。

英文摘要:

Inference accuracy of deep neural networks (DNNs) is a crucial performance metric, but can vary greatly in practice subject to actual test datasets and is typically unknown due to the lack of ground truth labels. This has raised significant concerns with trustworthiness of DNNs, especially in safety-critical applications. In this paper, we address trustworthiness of DNNs by using post-hoc processing to monitor the true inference accuracy on a user's dataset. Concretely, we propose a neural network-based accuracy monitor model, which only takes the deployed DNN's softmax probability output as its input and directly predicts if the DNN's prediction result is correct or not, thus leading to an estimate of the true inference accuracy. The accuracy monitor model can be pre-trained on a dataset relevant to the target application of interest, and only needs to actively label a small portion (1% in our experiments) of the user's dataset for model transfer. For estimation robustness, we further employ an ensemble of monitor models based on the Monte-Carlo dropout method. We evaluate our approach on different deployed DNN models for image classification and traffic sign detection over multiple datasets (including adversarial samples). The result shows that our accuracy monitor model provides a close-to-true accuracy estimation and outperforms the existing baseline methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑