arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2211.10892cs.CV
收藏

迈向现实的分布外检测:一种提高OOD检测泛化能力的新型评估框架

Towards Realistic Out-of-Distribution Detection: A Novel Evaluation Framework for Improving Generalization in OOD Detection

Vector Institute(向量研究所);Western University(韦仕敦大学);Okinawa Institute of Science and Technology(冲绳科学技术研究所)

Vahid Reza Khazaie, Anthony Wong, Mohammad Sabokrou

AI总结:

本文提出一种新型OOD检测评估框架,包含新数据集和泛化性评分,发现现有基准性能提升不代表现实可用性,并设计后处理阶段提升预训练模型在分布偏移下的表现。

摘要:

本文提出了一种新颖的分布外(OOD)检测评估框架,旨在更现实的场景中评估机器学习模型的性能。我们观察到,当前的测试协议无法满足现实世界中对OOD检测方法的测试需求。它们通常鼓励方法对正常数据的低多样性产生强烈偏见。为了解决这一局限性,我们提出了新的OOD测试数据集(CIFAR-10-R、CIFAR-100-R和ImageNet-30-R),使研究人员能够在现实分布偏移下对OOD检测性能进行基准测试。此外,我们引入了一个泛化性评分(GS)来衡量模型在OOD检测过程中的泛化能力。我们的实验表明,在现有基准数据集上提高性能并不一定能提高OOD检测模型在现实场景中的可用性。虽然利用深度预训练特征已被认为是OOD检测研究的一个有前景的方向,但我们的实验表明,在我们提出的数据集上测试的最先进的预训练模型性能显著下降。为了解决这个问题,我们提出了一个后处理阶段,在计算OOD分数之前调整预训练特征以适应这些分布偏移,这显著提高了最先进的预训练模型在我们基准上的性能。

英文摘要:

This paper presents a novel evaluation framework for Out-of-Distribution (OOD) detection that aims to assess the performance of machine learning models in more realistic settings. We observed that the real-world requirements for testing OOD detection methods are not satisfied by the current testing protocols. They usually encourage methods to have a strong bias towards a low level of diversity in normal data. To address this limitation, we propose new OOD test datasets (CIFAR-10-R, CIFAR-100-R, and ImageNet-30-R) that can allow researchers to benchmark OOD detection performance under realistic distribution shifts. Additionally, we introduce a Generalizability Score (GS) to measure the generalization ability of a model during OOD detection. Our experiments demonstrate that improving the performance on existing benchmark datasets does not necessarily improve the usability of OOD detection models in real-world scenarios. While leveraging deep pre-trained features has been identified as a promising avenue for OOD detection research, our experiments show that state-of-the-art pre-trained models tested on our proposed datasets suffer a significant drop in performance. To address this issue, we propose a post-processing stage for adapting pre-trained features under these distribution shifts before calculating the OOD scores, which significantly enhances the performance of state-of-the-art pre-trained models on our benchmarks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑