arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2209.11960cs.CVcs.LG
收藏

提高分布外检测评价标准

Raising the Bar on the Evaluation of Out-of-Distribution Detection

University of Oxford(牛津大学);Meta AI

Jishnu Mukhoti, Tsung-Yu Lin, Bor-Chun Chen, Ashish Shah, Philip H. S. Torr, Puneet K. Dokania, Ser-Nam Lim

摘要:

在图像分类中,对分布外数据的检测有了很大的发展。然而,大多数OoD检测方法都是在一组标准数据集上进行评估的,这些数据集与训练数据任意不同。没有明确的定义是什么形成了一个“好”的OoD数据集。此外,最先进的OoD检测方法已经在这些标准基准上实现了近乎完美的结果。在本文中,我们定义了2类OoD数据使用的感知/视觉和语义相似性的分布(iD)数据的微妙不同的概念。我们将近OoD样本定义为与iD样本在感知上相似但在语义上不同,而移位样本定义为视觉上不同但在语义上类似于iD数据的点。然后,我们提出了一个基于GAN的框架,用于在给定iD数据集的情况下,从这两个类别中的每一个类别生成OoD样本。通过在MNIST、CIFAR-10/100和ImageNet上进行的大量实验,我们表明:a)在传统基准测试中表现出色的最先进的OoD检测方法对我们提出的基准测试的鲁棒性明显降低。此外,b)在我们的设置上表现良好的模型在传统的现实世界OoD检测基准上也表现良好,反之亦然,从而表明人们甚至可能不需要单独的OoD集合来可靠地评估OoD检测的性能。

英文摘要:

In image classification, a lot of development has happened in detecting out-of-distribution (OoD) data. However, most OoD detection methods are evaluated on a standard set of datasets, arbitrarily different from training data. There is no clear definition of what forms a ``good" OoD dataset. Furthermore, the state-of-the-art OoD detection methods already achieve near perfect results on these standard benchmarks. In this paper, we define 2 categories of OoD data using the subtly different concepts of perceptual/visual and semantic similarity to in-distribution (iD) data. We define Near OoD samples as perceptually similar but semantically different from iD samples, and Shifted samples as points which are visually different but semantically akin to iD data. We then propose a GAN based framework for generating OoD samples from each of these 2 categories, given an iD dataset. Through extensive experiments on MNIST, CIFAR-10/100 and ImageNet, we show that a) state-of-the-art OoD detection methods which perform exceedingly well on conventional benchmarks are significantly less robust to our proposed benchmark. Moreover, b) models performing well on our setup also perform well on conventional real-world OoD detection benchmarks and vice versa, thereby indicating that one might not even need a separate OoD set, to reliably evaluate performance in OoD detection.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑