arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1812.05313cs.CV
收藏

当半监督学习遇到迁移学习:训练策略、模型和数据集

When Semi-Supervised Learning Meets Transfer Learning: Training Strategies, Models and Datasets

YouTu Lab, Tencent(腾讯优图实验室);Google Brain(谷歌大脑);Nanjing University(南京大学)

Hong-Yu Zhou, Avital Oliver, Jianxin Wu, Yefeng Zheng

摘要:

半监督学习(Semi-Supervised Learning,SSL)已被证明是同时利用标记和未标记数据的有效方法。最近的半监督方法专注于深度神经网络,并在几个基准上取得了可喜的成果:CIFAR 10,CIFAR 100和SVHN。然而,他们的大多数实验都是基于从头开始训练的模型,而不是预先训练的模型。另一方面,迁移学习在目标域具有有限标记数据时已经证明了其价值。这里有一个直观的问题:在微调预训练模型时,是否可以合并SSL?我们全面研究了从预训练模型开始的SSL方法在不同条件下的表现,包括训练策略,架构选择和数据集。从这项研究中,我们得到了一些有趣和有用的观察。虽然从业者对这些观察结果有直观的理解,但我们进行了全面的实证分析,并证明:(1)当从预训练模型训练时,SSL技术在完全监督基线上的增益比从随机初始化训练时小,(2)当用于训练预训练模型的源数据的域与目标任务的域显著不同时,来自SSL的收益显著更高,以及(3)一些SSL方法能够推进完全监督的基线(如伪标签)。我们希望我们的研究可以加深对SSL研究的理解,并促进开发更有效的SSL方法来利用预训练模型的过程。代码现在可以在github上找到。

英文摘要:

Semi-Supervised Learning (SSL) has been proved to be an effective way to leverage both labeled and unlabeled data at the same time. Recent semi-supervised approaches focus on deep neural networks and have achieved promising results on several benchmarks: CIFAR10, CIFAR100 and SVHN. However, most of their experiments are based on models trained from scratch instead of pre-trained models. On the other hand, transfer learning has demonstrated its value when the target domain has limited labeled data. Here comes the intuitive question: is it possible to incorporate SSL when fine-tuning a pre-trained model? We comprehensively study how SSL methods starting from pretrained models perform under varying conditions, including training strategies, architecture choice and datasets. From this study, we obtain several interesting and useful observations. While practitioners have had an intuitive understanding of these observations, we do a comprehensive emperical analysis and demonstrate that: (1) the gains from SSL techniques over a fully-supervised baseline are smaller when trained from a pre-trained model than when trained from random initialization, (2) when the domain of the source data used to train the pre-trained model differs significantly from the domain of the target task, the gains from SSL are significantly higher and (3) some SSL methods are able to advance fully-supervised baselines (like Pseudo-Label). We hope our studies can deepen the understanding of SSL research and facilitate the process of developing more effective SSL methods to utilize pre-trained models. Code is now available at github.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑