arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.27263cs.LG
收藏

ODP-Bench:分布外性能预测基准测试

ODP-Bench: Benchmarking Out-of-Distribution Performance Prediction

Department of Computer Science, Tsinghua University(清华大学计算机系);School of Information, Renmin University of China(中国人民大学信息学院)

Han Yu, Kehan Li, Dongbai Li, Yue He, Xingxuan Zhang, Peng Cui

AI总结:

针对分布外性能预测评估不一致且覆盖有限的问题,提出综合基准ODP-Bench,包含常用OOD数据集、现有算法及预训练模型,提供公平比较平台并深入分析算法能力边界。

摘要:

近年来,分布外(OOD)性能预测逐渐受到更多关注,其目标是预测已训练模型在无标签OOD测试数据集上的性能,以便在风险敏感场景中更好地利用和部署现成的已训练模型。尽管该领域取得了一定进展,但以往文献的评估协议不一致,且大多数工作仅涵盖有限数量的真实世界OOD数据集和分布偏移类型。为了为各种算法提供便捷且公平的比较,我们提出了分布外性能预测基准(ODP-Bench),这是一个包含最常用OOD数据集和现有实用性能预测算法的综合基准。我们提供已训练模型作为未来研究者的测试平台,从而保证比较的一致性并避免重复模型训练过程的负担。此外,我们还进行了深入的实验分析,以更好地理解它们的能力边界。

英文摘要:

Recently, there has been gradually more attention paid to Out-of-Distribution (OOD) performance prediction, whose goal is to predict the performance of trained models on unlabeled OOD test datasets, so that we could better leverage and deploy off-the-shelf trained models in risk-sensitive scenarios. Although progress has been made in this area, evaluation protocols in previous literature are inconsistent, and most works cover only a limited number of real-world OOD datasets and types of distribution shifts. To provide convenient and fair comparisons for various algorithms, we propose Out-of-Distribution Performance Prediction Benchmark (ODP-Bench), a comprehensive benchmark that includes most commonly used OOD datasets and existing practical performance prediction algorithms. We provide our trained models as a testbench for future researchers, thus guaranteeing the consistency of comparison and avoiding the burden of repeating the model training process. Furthermore, we also conduct in-depth experimental analyses to better understand their capability boundary.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑