探索并利用模型动物园中的多样化知识以实现域泛化
Explore and Exploit the Diverse Knowledge in Model Zoo for Domain Generalization
AI总结:
本文提出通过多样性偏移与相关性偏移刻画模型动物园中不同预训练模型的表示差异,并整合非仅最强模型的多样化知识,从而在多个数据集上提升分布外泛化并取得最先进结果。
摘要:
由于预训练技术的进步,预训练模型大量涌现,导致出现了一个包含大量公开可用模型的庞大模型动物园。有效利用这些资源,以为下游任务获得具有稳健分布外泛化能力的模型,已成为一个关键研究领域。以往研究主要集中于识别模型动物园中最强大的模型,而忽视了充分利用其中包含的多样化归纳偏置。本文认为较弱模型中包含的知识是有价值的,并提出了一种利用模型动物园内多样性来提升分布外泛化能力的方法。具体而言,我们通过从多样性偏移和相关性偏移两个维度刻画各种预训练模型在其编码表示上的变化,研究了它们在下游任务不同域上的行为。该刻画使我们能够提出一种整合多样化预训练模型的新算法,且整合对象不限于最强模型,从而实现更优的分布外泛化性能。我们提出的方法在多种数据集上展示了最先进的实证结果,从而验证了利用多样化知识的益处。
英文摘要:
The proliferation of pretrained models, as a result of advancements in pretraining techniques, has led to the emergence of a vast zoo of publicly available models. Effectively utilizing these resources to obtain models with robust out-of-distribution generalization capabilities for downstream tasks has become a crucial area of research. Previous research has primarily focused on identifying the most powerful models within the model zoo, neglecting to fully leverage the diverse inductive biases contained within. This paper argues that the knowledge contained in weaker models is valuable and presents a method for leveraging the diversity within the model zoo to improve out-of-distribution generalization capabilities. Specifically, we investigate the behaviors of various pretrained models across different domains of downstream tasks by characterizing the variations in their encoded representations in terms of two dimensions: diversity shift and correlation shift. This characterization enables us to propose a new algorithm for integrating diverse pretrained models, not limited to the strongest models, in order to achieve enhanced out-of-distribution generalization performance. Our proposed method demonstrates state-of-the-art empirical results on a variety of datasets, thus validating the benefits of utilizing diverse knowledge.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。