只需旋转它!通过多次查询对闭源模型进行不确定性估计
Just rotate it! Uncertainty estimation in closed-source models via multiple queries
AI总结:
提出一种通过多次查询变换图像来估计闭源模型不确定性的方法,发现自然变换优于高斯扰动,并利用迁移学习进一步改善校准。
摘要:
我们提出了一种简单有效的方法来估计闭源深度神经网络图像分类模型的不确定性。给定一张基础图像,我们的方法创建多个变换版本,并使用它们查询闭源模型的top-1预测。与将所有预测赋予100%置信度的朴素基线相比,我们展示了不确定性估计校准的显著改进。虽然我们最初探索了高斯扰动,但我们的实证发现表明,自然变换(如旋转和弹性变形)能产生校准更好的预测。此外,通过实证结果和直接的理论分析,我们阐明了自然变换优于高斯噪声的原因。利用这些见解,我们提出了一种迁移学习方法,进一步改善了我们的校准结果。
英文摘要:
We propose a simple and effective method to estimate the uncertainty of closed-source deep neural network image classification models. Given a base image, our method creates multiple transformed versions and uses them to query the top-1 prediction of the closed-source model. We demonstrate significant improvements in the calibration of uncertainty estimates compared to the naive baseline of assigning 100\% confidence to all predictions. While we initially explore Gaussian perturbations, our empirical findings indicate that natural transformations, such as rotations and elastic deformations, yield even better-calibrated predictions. Furthermore, through empirical results and a straightforward theoretical analysis, we elucidate the reasons behind the superior performance of natural transformations over Gaussian noise. Leveraging these insights, we propose a transfer learning approach that further improves our calibration results.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。