ZooD：利用Model Zoo实现分布外泛化
ZooD: Exploiting Model Zoo for Out-of-Distribution Generalization
摘要：
大规模预训练的最新进展已经显示出利用大量预训练模型（PTM）来改善分布外（OoD）泛化的巨大潜力，其目标是在对多个训练域进行微调后，在可能的未知域上表现良好。然而，最大限度地利用PTM的动物园是具有挑战性的，因为微调PTM的所有可能组合在计算上是禁止的，而PTM的准确选择需要解决OoD任务的可能数据分布偏移。在这项工作中，我们提出了ZooD，PTM排名和集成与特征选择的范例。我们提出的度量排名PTM通过量化类间区分度和域间稳定性的功能提取的PTM在留一域交叉验证的方式。然后为目标OoD任务聚合排名前K的模型。为了避免模型集成引起的噪声积累，我们提出了一种有效的变分EM算法来选择信息特征。我们在一个由35个模型组成的不同模型动物园上评估了我们的范例，并证明了：（i）模型排名与微调排名的相关性比以前的方法更好，比暴力微调快9859倍;（ii）具有特征选择的模型集成后的OoD泛化优于当前状态，在最具挑战性的任务DomainNet上，准确率从46.5%提高到50.6%。此外，我们还提供了35个PTM在7个OoD数据集上的微调结果，希望对模型动物园和OoD泛化的研究有所帮助。代码将在https://gitee.com/mindspore/models/tree/master/research/cv/zood上提供。
英文摘要：
Recent advances on large-scale pre-training have shown great potentials of leveraging a large set of Pre-Trained Models (PTMs) for improving Out-of-Distribution (OoD) generalization, for which the goal is to perform well on possible unseen domains after fine-tuning on multiple training domains. However, maximally exploiting a zoo of PTMs is challenging since fine-tuning all possible combinations of PTMs is computationally prohibitive while accurate selection of PTMs requires tackling the possible data distribution shift for OoD tasks. In this work, we propose ZooD, a paradigm for PTMs ranking and ensemble with feature selection. Our proposed metric ranks PTMs by quantifying inter-class discriminability and inter-domain stability of the features extracted by the PTMs in a leave-one-domain-out cross-validation manner. The top-K ranked models are then aggregated for the target OoD task. To avoid accumulating noise induced by model ensemble, we propose an efficient variational EM algorithm to select informative features. We evaluate our paradigm on a diverse model zoo consisting of 35 models for various OoD tasks and demonstrate: (i) model ranking is better correlated with fine-tuning ranking than previous methods and up to 9859x faster than brute-force fine-tuning; (ii) OoD generalization after model ensemble with feature selection outperforms the state-of-the-art methods and the accuracy on most challenging task DomainNet is improved from 46.5\% to 50.6\%. Furthermore, we provide the fine-tuning results of 35 PTMs on 7 OoD datasets, hoping to help the research of model zoo and OoD generalization. Code will be available at https://gitee.com/mindspore/models/tree/master/research/cv/zood.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
