SMILE:自蒸馏混合,用于高效的迁移学习
SMILE: Self-Distilled MIxup for Efficient Transfer LEarning
摘要:
为了提高深度学习的性能,mixup被提出来迫使神经网络在训练样本之间倾向于简单的线性行为。然而,使用预训练模型执行迁移学习的混合并不是那么简单,具有大型全连接(FC)层的高容量预训练模型很容易过拟合到目标数据集,即使样本到标签混合。在这项工作中,我们提出了SMILE -自蒸馏混合有效的迁移学习。以混合图像作为输入,SMILE正则化CNN特征提取器的输出,以从输入的混合特征向量(样本到特征的混合)以及混合标签中学习。具体来说,SMILE结合了一个从预训练模型继承的平均教师,以自蒸馏的方式提供输入样本的特征向量,并通过一个新的三元组正则化器相应地混合特征向量。三重正则化器平衡了特征空间和标签空间中的混合效应,同时限制了预训练任务样本之间的线性度。已经进行了大量的实验来验证SMILE所做的性能改进,与广泛的迁移学习算法进行比较,包括微调,L2-SP,DELTA和RIFLE,甚至与混合策略相结合。消融研究表明,香草样本到标签的混合策略可以稍微增加训练样本之间的线性,但缺乏泛化能力,而SMILE显着改善了训练和测试数据集的标签和特征空间的混合效果。经验观察支持我们的设计直觉和目的。
英文摘要:
To improve the performance of deep learning, mixup has been proposed to force the neural networks favoring simple linear behaviors in-between training samples. Performing mixup for transfer learning with pre-trained models however is not that simple, a high capacity pre-trained model with a large fully-connected (FC) layer could easily overfit to the target dataset even with samples-to-labels mixed up. In this work, we propose SMILE - Self-Distilled Mixup for EffIcient Transfer LEarning. With mixed images as inputs, SMILE regularizes the outputs of CNN feature extractors to learn from the mixed feature vectors of inputs (sample-to-feature mixup), in addition to the mixed labels. Specifically, SMILE incorporates a mean teacher, inherited from the pre-trained model, to provide the feature vectors of input samples in a self-distilling fashion, and mixes up the feature vectors accordingly via a novel triplet regularizer. The triple regularizer balances the mixup effects in both feature and label spaces while bounding the linearity in-between samples for pre-training tasks. Extensive experiments have been done to verify the performance improvement made by SMILE, in comparisons with a wide spectrum of transfer learning algorithms, including fine-tuning, L2-SP, DELTA, and RIFLE, even with mixup strategies combined. Ablation studies show that the vanilla sample-to-label mixup strategies could marginally increase the linearity in-between training samples but lack of generalizability, while SMILE significantly improve the mixup effects in both label and feature spaces with both training and testing datasets. The empirical observations backup our design intuition and purposes.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。