使用SimCLR的半监督学习、迁移学习和知识蒸馏
Semi-Supervising Learning, Transfer Learning, and Knowledge Distillation with SimCLR
摘要：
最近在半监督学习领域的突破已经取得了与最先进的传统监督学习方法相匹配的结果。计算机视觉中最成功的半监督学习方法集中在利用大量未标记的数据，通过数据增强和转换学习一般表示，创建伪标签，实现不同的损失函数，并最终将这些知识转移到更特定于任务的较小模型。在本文中，我们的目标是进行我们的分析，在三个不同的方面Simplified，目前国家的最先进的半监督学习框架计算机视觉。首先，我们分析了对比学习在微调上的特性，因为我们知道对比学习是这种方法如此成功的原因。第二，通过教师强制范式研究知识蒸馏。我们观察到当教师和学生共享相同的基础模型时，知识蒸馏会取得更好的效果。最后，我们研究了迁移学习是如何工作的，以及它与不同数据集上的类数之间的关系。我们的研究结果表明，迁移学习表现得更好时，类的数量较少。
英文摘要：
Recent breakthroughs in the field of semi-supervised learning have achieved results that match state-of-the-art traditional supervised learning methods. Most successful semi-supervised learning approaches in computer vision focus on leveraging huge amount of unlabeled data, learning the general representation via data augmentation and transformation, creating pseudo labels, implementing different loss functions, and eventually transferring this knowledge to more task-specific smaller models. In this paper, we aim to conduct our analyses on three different aspects of SimCLR, the current state-of-the-art semi-supervised learning framework for computer vision. First, we analyze properties of contrast learning on fine-tuning, as we understand that contrast learning is what makes this method so successful. Second, we research knowledge distillation through teacher-forcing paradigm. We observe that when the teacher and the student share the same base model, knowledge distillation will achieve better result. Finally, we study how transfer learning works and its relationship with the number of classes on different data sets. Our results indicate that transfer learning performs better when number of classes are smaller.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
