We propose a novel approach for estimating the difficulty and transferability of supervised classification tasks. Unlike previous work, our approach is solution agnostic and does not require or assume trained models. Instead, we estimate these values using an information theoretic approach: treating training labels as random variables and exploring their statistics. When transferring from a source to a target task, we consider the conditional entropy between two such variables (i.e., label assignments of the two tasks). We show analytically and empirically that this value is related to the loss of the transferred model. We further show how to use this value to estimate task hardness. We test our claims extensively on three large scale data sets -- CelebA (40 tasks), Animals with Attributes 2 (85 tasks), and Caltech-UCSD Birds 200 (312 tasks) -- together representing 437 classification tasks. We provide results showing that our hardness and transferability estimates are strongly correlated with empirical hardness and transferability. As a case study, we transfer a learned face recognition model to CelebA attribute classification tasks, showing state of the art accuracy for tasks estimated to be highly transferable.
Improving Transferability of Representations via Augmentation-Aware Self-Supervision
Korea Advanced Institute of Science and Technology (KAIST)(韩国科学技术院);University of Michigan(密歇根大学);University of California, Berkeley(加州大学伯克利分校);Amazon Web Services(亚马逊云科技);LG AI Research(LG人工智能研究院)
voice2mode: Phonation Mode Classification in Singing using Self-Supervised Speech Models
University of Birmingham, School of Computer Science, Birmingham, UK(伯明翰大学计算机科学学院);Carnegie Mellon University, Information Systems, Doha, Qatar(卡内基梅隆大学信息系统);University of Southern California, Department of Electrical(南加州大学电气工程系)