分布外检测应使用保形预测(反之亦然?)
Out-of-Distribution Detection Should Use Conformal Prediction (and Vice-versa?)
AI总结:
本文提出将保形预测用于分布外检测评分评估,定义保形AUROC和FRP@TPR95指标以提供保守性保证,并在OpenOOD和ADBench基准上验证,同时反向利用OOD评分改进CP方法,揭示两领域内在关联。
摘要:
分布外(OOD)检测的研究主要集中于构建能有效区分OOD数据与分布内(ID)数据的评分。另一方面,保形预测(CP)使用非一致性评分来构建具有概率覆盖保证的预测集。在本工作中,我们提出使用CP来更好地评估OOD评分的效率。具体而言,我们强调在标准OOD基准设置中,由于测试数据集的有限样本量,评估指标可能过于乐观。基于(Bates等人,2022)的工作,我们定义了新的保形AUROC和保形FRP@TPR95指标,这些指标是修正项,为这些指标的变异性提供概率保守性保证。我们展示了这些修正对两个参考OOD和异常检测基准的影响,即OpenOOD(Yang等人,2022)和ADBench(Han等人,2022)。我们还表明,将OOD与CP结合使用的益处反过来也适用,即使用OOD评分作为非一致性评分,从而改进了当前的CP方法。这些贡献的关键信息之一是,由于OOD关注于设计评分,而CP关注于解释这些评分,这两个领域可能本质上相互交织。
英文摘要:
Research on Out-Of-Distribution (OOD) detection focuses mainly on building scores that efficiently distinguish OOD data from In Distribution (ID) data. On the other hand, Conformal Prediction (CP) uses non-conformity scores to construct prediction sets with probabilistic coverage guarantees. In this work, we propose to use CP to better assess the efficiency of OOD scores. Specifically, we emphasize that in standard OOD benchmark settings, evaluation metrics can be overly optimistic due to the finite sample size of the test dataset. Based on the work of (Bates et al., 2022), we define new conformal AUROC and conformal FRP@TPR95 metrics, which are corrections that provide probabilistic conservativeness guarantees on the variability of these metrics. We show the effect of these corrections on two reference OOD and anomaly detection benchmarks, OpenOOD (Yang et al., 2022) and ADBench (Han et al., 2022). We also show that the benefits of using OOD together with CP apply the other way around by using OOD scores as non-conformity scores, which results in improving upon current CP methods. One of the key messages of these contributions is that since OOD is concerned with designing scores and CP with interpreting these scores, the two fields may be inherently intertwined.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。