Assessing Pre-Trained Models for Transfer Learning Through Distribution of Spectral Components
School of Data Science and Engineering, East China Normal University(华东师范大学数据科学与工程学院);School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳)计算机科学与技术学院)
Tengxue Zhang, Yang Shu, Xinyang Chen, Yifei Long, Chenjuan Guo, Bin Yang
Pre-trained model assessment for transfer learning aims to identify the optimal candidate for the downstream tasks from a model hub, without the need of time-consuming fine-tuning. Existing advanced works mainly focus on analyzing the intrinsic characteristics of the entire features extracted by each pre-trained model or how well such features fit the target labels. This paper proposes a novel perspective for pre-trained model assessment through the Distribution of Spectral Components (DISCO). Through singular value decomposition of features extracted from pre-trained models, we investigate different spectral components and observe that they possess distinct transferability, contributing diversely to the fine-tuning performance. Inspired by this, we propose an assessment method based on the distribution of spectral components which measures the proportions of their corresponding singular values. Pre-trained models with features concentrating on more transferable components are regarded as better choices for transfer learning. We further leverage the labels of downstream data to better estimate the transferability of each spectral component and derive the final assessment criterion. Our proposed method is flexible and can be applied to both classification and regression tasks. We conducted comprehensive experiments across three benchmarks and two tasks including image classification and object detection, demonstrating that our method achieves state-of-the-art performance in choosing proper pre-trained models from the model hub for transfer learning.
LogME: Practical Assessment of Pre-trained Models for Transfer Learning
BNRist(北京国家信息科学与技术研究中心);Tsinghua University(清华大学)
本文研究任务自适应预训练模型选择,这是一个尚未充分研究的问题,即评估目标任务的预训练模型并从模型动物园\emph{without fine-tuning}中选择最佳模型。一些试点工作解决了将有监督的预训练模型转移到分类任务中的问题,但它们无法处理新出现的无监督的预训练模型或回归任务。为了寻求一种实用的评估方法,我们提出了估计标签证据的最大值,给出了由预训练模型提取的特征。与最大似然不同,最大证据是\emph{immune to over-fitting},而其昂贵的计算可以通过我们精心设计的算法大大减少。最大证据对数(LogME)可用于评估迁移学习的预训练模型:具有高LogME值的预训练模型可能具有良好的迁移性能。LogME是\emph{fast, accurate, and general},它是评估预训练模型的第一种实用方法。与暴力微调相比,LogME最多在挂钟时间内带来$3000\times$加速,并且只需要$1\%$内存占用。它在其设置中大大优于先前的方法,并且适用于新的设置。它对于不同的预训练模型(有监督的预训练和无监督的预训练)、下游任务(分类和回归)和模式(视觉和语言)来说足够通用。代码可在以下存储库中获得:\href{https://github.com/thuml/LogME}{https://github.com/thuml/LogME}。
CLOSURE: Assessing Systematic Generalization of CLEVR Models
Element AI(元素人工智能公司);Quebec Artificial Intelligence Institute (Mila)(魁北克人工智能研究所(米拉));Université de Montréal(蒙特利尔大学);McGill University(麦吉尔大学);Stanford University(斯坦福大学)
Assessing Reusability of Deep Learning-Based Monotherapy Drug Response Prediction Models Trained with Omics Data
Argonne National Laboratory(阿贡国家实验室);The University of Chicago(芝加哥大学);Frederick National Laboratory for Cancer Research(弗雷德里克国家癌症研究实验室);Mayo Clinic(梅奥诊所)
Assessing the Generalizability of a Performance Predictive Model
Jožef Stefan Institute(约瑟夫·斯特凡研究所);Jožef Stefan International Postgraduate School(约瑟夫·斯特凡国际研究生院);Leiden University(莱顿大学);Stellenbosch University(斯泰伦博斯大学);Gulf University for Science and Technology(海湾科技大学);Sorbonne Université(索邦大学);CNRS(法国国家科学研究中心)
Deconfounding age effects with fair representation learning when assessing dementia
Winterlight Labs;University of Toronto(多伦多大学);Li Ka Shing Knowledge Institute, St Michael’s Hospital(圣迈克尔医院李嘉诚知识研究所);Vector Institute for Artificial Intelligence(Vector人工智能研究所);Surgical Safety Technologies Inc(Surgical Safety Technologies公司)