In this paper, we propose that small models may not need to absorb the cost of pre-training to reap its benefits. Instead, they can capitalize on the astonishing results achieved by modern, enormous models to a surprising degree. We observe that, when distilled on a task from a pre-trained teacher model, a small model can achieve or surpass the performance it would achieve if it was pre-trained then finetuned on that task. To allow this phenomenon to be easily leveraged, we establish a connection reducing knowledge distillation to modern contrastive learning, opening two doors: (1) vastly different model architecture pairings can work for the distillation, and (2) most contrastive learning algorithms rooted in the theory of Noise Contrastive Estimation can be easily applied and used. We demonstrate this paradigm using pre-trained teacher models from open-source model hubs, Transformer and convolution based model combinations, and a novel distillation algorithm that massages the Alignment/Uniformity perspective of contrastive learning by Wang & Isola (2020) into a distillation objective. We choose this flavor of contrastive learning due to its low computational cost, an overarching theme of this work. We also observe that this phenomenon tends not to occur if the task is data-limited. However, this can be alleviated by leveraging yet another scale-inspired development: large, pre-trained generative models for dataset augmentation. Again, we use an open-source model, and our rudimentary prompts are sufficient to boost the small model`s performance. Thus, we highlight a training method for small models that is up to 94% faster than the standard pre-training paradigm without sacrificing performance. For practitioners discouraged from fully utilizing modern foundation datasets for their small models due to the prohibitive scale, we believe our work keeps that door open.
Dept. Computing Science & Alberta Machine Intelligence Institute (Amii), University of Alberta(计算科学系及阿尔伯塔人工智能研究所(Amii),阿尔伯塔大学);Canada CIFAR AI Chair, Amii(加拿大CIFAR人工智能主席,Amii)
Spectral Co-Distillation for Personalized Federated Learning
Singapore University of Technology and Design (SUTD)(新加坡科技设计大学);Zhejiang University/University of Illinois Urbana-Champaign Institute, Zhejiang University(浙江大学伊利诺伊大学厄巴纳-香槟校区联合学院,浙江大学)
Beyond Classification: Knowledge Distillation using Multi-Object Impressions
Indian Institute of Science(印度科学理工学院)
知识蒸馏(KD)利用训练数据作为传递集将知识从复杂网络(教师)传递到较小网络(学生)。一些作品最近已经确定了许多情况下,训练数据可能是不可用的,由于数据隐私或敏感性的问题,并提出了解决方案,在此限制性约束的分类任务。与现有的作品不同,我们第一次解决了一个更具挑战性的问题,即,“KD for目标检测with zero knowledge about the训练数据及其统计”.我们提出的方法只使用预训练的Faster RCNN Teacher网络来准备伪目标并合成相应的样本(称为“多对象immersing”)。我们使用这个伪数据集作为一个转移集进行zero-shot KD的目标检测。我们证明了我们提出的方法的有效性,通过几个消融和广泛的实验基准数据集,如KITTI,Pascal和COCO。我们的方法没有训练样本,在具有相同和一半容量的学生上实现了64.2%和55.5%的可观mAP,同时在KITTI上从73.3% mAP的Resnet-18教师进行蒸馏。
TriDeNT: Triple Deep Network Training for Privileged Knowledge Distillation in Histopathology
School of Cancer Sciences, University of Glasgow(格拉斯哥大学癌症科学学院);Cancer Research UK Scotland Institute(英国癌症研究苏格兰研究所);School of Computing Science, University of Glasgow(格拉斯哥大学计算科学学院);School of Mathematics and Statistics, University of Glasgow(格拉斯哥大学数学与统计学院);Division of Biosciences, University College London(伦敦大学学院生物科学部)