多级知识蒸馏与动态自监督学习用于持续学习
Multi-Level Knowledge Distillation and Dynamic Self-Supervised Learning for Continual Learning
AI总结：
本文提出多级知识蒸馏和动态自监督学习，用于解决持续学习中的类别增量问题，通过利用未标记数据提升模型稳定性与可塑性，实验显示在CVPR CLVISION挑战中取得第二名。
摘要：
类别增量与重复（CIR）是一种比传统类别增量更现实的场景，假设先前训练的类别在后续任务中重复出现。CIR假设可以轻松访问大量未标记数据，如互联网。因此，我们提出两个组件，高效利用未标记数据以确保CIR设置下模型的高稳定性和可塑性。首先，我们引入多级知识蒸馏（MLKD），从多个先前模型中蒸馏知识，包括特征和logits，使模型能保持多种先前知识。此外，我们实现动态自监督损失（SSL）以利用未标记数据加速新类学习，动态加权SSL保持训练重点在主要任务上。所提组件显著提升了CIR设置的性能，在CVPR 5th CLVISION挑战中取得第二名。
英文摘要：
Class-incremental with repetition (CIR), where previously trained classes repeatedly introduced in future tasks, is a more realistic scenario than the traditional class incremental setup, which assumes that each task contains unseen classes. CIR assumes that we can easily access abundant unlabeled data from external sources, such as the Internet. Therefore, we propose two components that efficiently use the unlabeled data to ensure the high stability and the plasticity of models trained in CIR setup. First, we introduce multi-level knowledge distillation (MLKD) that distills knowledge from multiple previous models across multiple perspectives, including features and logits, so the model can maintain much various previous knowledge. Moreover, we implement dynamic self-supervised loss (SSL) to utilize the unlabeled data that accelerates the learning of new classes, while dynamic weighting of SSL keeps the focus of training to the primary task. Both of our proposed components significantly improve the performance in CIR setup, achieving 2nd place in the CVPR 5th CLVISION Challenge.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
