arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2412.19085cs.LG
收藏

通过频谱成分分布评估预训练模型在迁移学习中的表现

Assessing Pre-Trained Models for Transfer Learning Through Distribution of Spectral Components

School of Data Science and Engineering, East China Normal University(华东师范大学数据科学与工程学院);School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳)计算机科学与技术学院)

Tengxue Zhang, Yang Shu, Xinyang Chen, Yifei Long, Chenjuan Guo, Bin Yang

AI总结:

本文提出基于频谱成分分布(DISCO)的预训练模型评估方法,通过奇异值分解分析特征成分的迁移能力,在分类和回归任务及三大基准上达到最先进的模型选择性能。

摘要:

迁移学习中的预训练模型评估旨在从模型库中为下游任务确定最佳候选模型,而无需进行耗时的微调。现有的先进工作主要集中在分析每个预训练模型提取的整个特征的内在特性,或者这些特征与目标标签的契合程度。本文通过频谱成分分布(DISCO)提出了一种新的预训练模型评估视角。通过对预训练模型提取的特征进行奇异值分解,我们研究了不同的频谱成分,并观察到它们具有不同的迁移能力,对微调性能的贡献各不相同。受此启发,我们提出了一种基于频谱成分分布的评估方法,该方法测量其对应奇异值的比例。特征集中在更具迁移性成分上的预训练模型被认为是迁移学习的更好选择。我们进一步利用下游数据的标签来更好地估计每个频谱成分的迁移能力,并得出最终的评估标准。我们提出的方法具有灵活性,可应用于分类和回归任务。我们在三个基准和两项任务(包括图像分类和目标检测)上进行了全面实验,证明我们的方法在从模型库中选择适合迁移学习的预训练模型方面达到了最先进的性能。

英文摘要:

Pre-trained model assessment for transfer learning aims to identify the optimal candidate for the downstream tasks from a model hub, without the need of time-consuming fine-tuning. Existing advanced works mainly focus on analyzing the intrinsic characteristics of the entire features extracted by each pre-trained model or how well such features fit the target labels. This paper proposes a novel perspective for pre-trained model assessment through the Distribution of Spectral Components (DISCO). Through singular value decomposition of features extracted from pre-trained models, we investigate different spectral components and observe that they possess distinct transferability, contributing diversely to the fine-tuning performance. Inspired by this, we propose an assessment method based on the distribution of spectral components which measures the proportions of their corresponding singular values. Pre-trained models with features concentrating on more transferable components are regarded as better choices for transfer learning. We further leverage the labels of downstream data to better estimate the transferability of each spectral component and derive the final assessment criterion. Our proposed method is flexible and can be applied to both classification and regression tasks. We conducted comprehensive experiments across three benchmarks and two tasks including image classification and object detection, demonstrating that our method achieves state-of-the-art performance in choosing proper pre-trained models from the model hub for transfer learning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑