LogME:迁移学习预训练模型的实用评估
LogME: Practical Assessment of Pre-trained Models for Transfer Learning
摘要:
本文研究任务自适应预训练模型选择,这是一个尚未充分研究的问题,即评估目标任务的预训练模型并从模型动物园\emph{without fine-tuning}中选择最佳模型。一些试点工作解决了将有监督的预训练模型转移到分类任务中的问题,但它们无法处理新出现的无监督的预训练模型或回归任务。为了寻求一种实用的评估方法,我们提出了估计标签证据的最大值,给出了由预训练模型提取的特征。与最大似然不同,最大证据是\emph{immune to over-fitting},而其昂贵的计算可以通过我们精心设计的算法大大减少。最大证据对数(LogME)可用于评估迁移学习的预训练模型:具有高LogME值的预训练模型可能具有良好的迁移性能。LogME是\emph{fast, accurate, and general},它是评估预训练模型的第一种实用方法。与暴力微调相比,LogME最多在挂钟时间内带来$3000\times$加速,并且只需要$1\%$内存占用。它在其设置中大大优于先前的方法,并且适用于新的设置。它对于不同的预训练模型(有监督的预训练和无监督的预训练)、下游任务(分类和回归)和模式(视觉和语言)来说足够通用。代码可在以下存储库中获得:\href{https://github.com/thuml/LogME}{https://github.com/thuml/LogME}。
英文摘要:
This paper studies task adaptive pre-trained model selection, an underexplored problem of assessing pre-trained models for the target task and select best ones from the model zoo \emph{without fine-tuning}. A few pilot works addressed the problem in transferring supervised pre-trained models to classification tasks, but they cannot handle emerging unsupervised pre-trained models or regression tasks. In pursuit of a practical assessment method, we propose to estimate the maximum value of label evidence given features extracted by pre-trained models. Unlike the maximum likelihood, the maximum evidence is \emph{immune to over-fitting}, while its expensive computation can be dramatically reduced by our carefully designed algorithm. The Logarithm of Maximum Evidence (LogME) can be used to assess pre-trained models for transfer learning: a pre-trained model with a high LogME value is likely to have good transfer performance. LogME is \emph{fast, accurate, and general}, characterizing itself as the first practical method for assessing pre-trained models. Compared with brute-force fine-tuning, LogME brings at most $3000\times$ speedup in wall-clock time and requires only $1\%$ memory footprint. It outperforms prior methods by a large margin in their setting and is applicable to new settings. It is general enough for diverse pre-trained models (supervised pre-trained and unsupervised pre-trained), downstream tasks (classification and regression), and modalities (vision and language). Code is available at this repository: \href{https://github.com/thuml/LogME}{https://github.com/thuml/LogME}.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。