arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.03074cs.LG
收藏

MetaOOD:OOD检测模型的自动选择

MetaOOD: Automatic Selection of OOD Detection Models

University of Southern California(南加州大学);University of Chicago(芝加哥大学);Carnegie Mellon University(卡内基梅隆大学)

Yuehan Qin, Yichi Zhang, Yi Nian, Xueying Ding, Yue Zhao

AI总结:

MetaOOD提出首个零样本无监督元学习框架,利用历史性能数据和语言模型嵌入自动选择OOD检测模型,在24个数据集对上显著优于11个基线,且时间开销极小。

摘要:

我们如何为各种底层任务自动选择分布外(OOD)检测模型?这对于通过识别数据分布偏移来维护开放世界应用的可靠性至关重要,特别是在在线交易、自动驾驶和实时患者诊断等关键领域。尽管存在众多OOD检测方法,但为不同任务选择最优模型的挑战在很大程度上仍未得到充分探索,尤其是在缺乏真实标签的场景中。在这项工作中,我们引入了MetaOOD,这是第一个利用元学习自动选择OOD检测模型的零样本、无监督框架。作为一种元学习方法,MetaOOD利用现有方法在各种基准OOD检测数据集上的历史性能数据,从而能够在测试时无需标记数据的情况下,为新数据集有效选择合适的模型。为了更准确地量化任务相似性,我们引入了基于语言模型的嵌入,以捕捉数据集和检测模型的独特OOD特征。通过对24个独特的测试数据集对进行广泛实验,从11个OOD检测模型中进行选择,我们证明了MetaOOD显著优于现有方法,并且仅带来微小的额外时间开销。我们的结果经Wilcoxon统计检验验证,表明MetaOOD超越了包括成熟OOD检测器和先进无监督选择方法在内的11个不同基线。

英文摘要:

How can we automatically select an out-of-distribution (OOD) detection model for various underlying tasks? This is crucial for maintaining the reliability of open-world applications by identifying data distribution shifts, particularly in critical domains such as online transactions, autonomous driving, and real-time patient diagnosis. Despite the availability of numerous OOD detection methods, the challenge of selecting an optimal model for diverse tasks remains largely underexplored, especially in scenarios lacking ground truth labels. In this work, we introduce MetaOOD, the first zero-shot, unsupervised framework that utilizes meta-learning to select an OOD detection model automatically. As a meta-learning approach, MetaOOD leverages historical performance data of existing methods across various benchmark OOD detection datasets, enabling the effective selection of a suitable model for new datasets without the need for labeled data at the test time. To quantify task similarities more accurately, we introduce language model-based embeddings that capture the distinctive OOD characteristics of both datasets and detection models. Through extensive experimentation with 24 unique test dataset pairs to choose from among 11 OOD detection models, we demonstrate that MetaOOD significantly outperforms existing methods and only brings marginal time overhead. Our results, validated by Wilcoxon statistical tests, show that MetaOOD surpasses a diverse group of 11 baselines, including established OOD detectors and advanced unsupervised selection methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑