arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2109.05554cs.LG
收藏

没有真正的艺术?数据集之间的OOD检测方法不一致

No True State-of-the-Art? OOD Detection Methods are Inconsistent across Datasets

Stanford University(斯坦福大学)

Fahim Tajwar, Ananya Kumar, Sang Michael Xie, Percy Liang

摘要:

分布外检测是可靠ML系统的重要组成部分。现有文献已经提出了各种方法(例如,MSP(Hendrycks & Gimpel,2017),ODIN(Liang等人,2018)、Mahalanobis(Lee等人,2018)),声称它们是最先进的,通过显示它们在选定的一组分布内(ID)和分布外(OOD)数据集上优于以前的方法。在这项工作中,我们表明,没有这些方法是固有的更好地在OOD检测比其他标准化的一组16(ID,OOD)对。我们给出了可能的解释,这些不一致的简单的玩具数据集,其中一种方法是否优于另一种方法取决于结构的ID和OOD数据集的问题。最后,我们表明,一个方法优于另一个在一定的(ID,OOD)对可能不会这样做,在低数据制度。在低数据的制度,我们提出了一种基于距离的方法,成对OOD检测(POD),这是基于暹罗网络和Mahalanobis通过回避昂贵的协方差估计步骤。我们的研究结果表明,OOD检测问题可能过于广泛,我们应该考虑更具体的杠杆结构。

英文摘要:

Out-of-distribution detection is an important component of reliable ML systems. Prior literature has proposed various methods (e.g., MSP (Hendrycks & Gimpel, 2017), ODIN (Liang et al., 2018), Mahalanobis (Lee et al., 2018)), claiming they are state-of-the-art by showing they outperform previous methods on a selected set of in-distribution (ID) and out-of-distribution (OOD) datasets. In this work, we show that none of these methods are inherently better at OOD detection than others on a standardized set of 16 (ID, OOD) pairs. We give possible explanations for these inconsistencies with simple toy datasets where whether one method outperforms another depends on the structure of the ID and OOD datasets in question. Finally, we show that a method outperforming another on a certain (ID, OOD) pair may not do so in a low-data regime. In the low-data regime, we propose a distance-based method, Pairwise OOD detection (POD), which is based on Siamese networks and improves over Mahalanobis by sidestepping the expensive covariance estimation step. Our results suggest that the OOD detection problem may be too broad, and we should consider more specific structures for leverage.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑