面向鲁棒半监督持续学习的动态子图蒸馏
Dynamic Sub-graph Distillation for Robust Semi-supervised Continual Learning
AI总结：
针对半监督持续学习中未标注数据分布不可靠导致训练不稳定、性能受影响的问题，提出动态子图蒸馏方法，结合语义与结构信息实现稳定蒸馏，适配多种场景，经多数据集验证可有效缓解灾难性遗忘。
摘要：
持续学习（CL）已取得可观成果，性能与全监督模式下的一次性学习相当。但持续学习策略通常需要大量标注样本，这给其实际部署带来挑战。本文聚焦半监督持续学习（SSCL），即模型从类别未知的部分标注数据中逐步学习的场景。我们对半监督持续学习进行了全面分析，证明未标注数据的不可靠分布会导致学习进展阶段的训练与优化不稳定，该问题会严重影响半监督持续学习的性能。为解决这一局限，我们提出一种名为动态子图蒸馏（DSGD）的半监督持续学习新方法，该方法同时利用语义与结构信息，在未标注数据上实现更稳定的知识蒸馏，且对分布偏差具备鲁棒性。首先，我们形式化了结构蒸馏的通用模型，并为持续学习过程设计了动态图构建机制。接着，我们定义了结构蒸馏向量，设计了动态子图蒸馏算法，该算法支持端到端训练，且具备适配任务规模扩展的能力。整个方法可适配多种持续学习方法与监督设置。最后，我们在CIFAR10、CIFAR100和ImageNet-100三个数据集上，采用不同监督比例开展实验，结果证明所提方法能有效缓解半监督持续学习场景中的灾难性遗忘问题。
英文摘要：
Continual learning (CL) has shown promising results and comparable performance to learning at once in a fully supervised manner. However, CL strategies typically require a large number of labeled samples, making their real-life deployment challenging. In this work, we focus on semi-supervised continual learning (SSCL), where the model progressively learns from partially labeled data with unknown categories. We provide a comprehensive analysis of SSCL and demonstrate that unreliable distributions of unlabeled data lead to unstable training and refinement of the progressing stages. This problem severely impacts the performance of SSCL. To address the limitations, we propose a novel approach called Dynamic Sub-Graph Distillation (DSGD) for semi-supervised continual learning, which leverages both semantic and structural information to achieve more stable knowledge distillation on unlabeled data and exhibit robustness against distribution bias. Firstly, we formalize a general model of structural distillation and design a dynamic graph construction for the continual learning progress. Next, we define a structure distillation vector and design a dynamic sub-graph distillation algorithm, which enables end-to-end training and adaptability to scale up tasks. The entire proposed method is adaptable to various CL methods and supervision settings. Finally, experiments conducted on three datasets CIFAR10, CIFAR100, and ImageNet-100, with varying supervision ratios, demonstrate the effectiveness of our proposed approach in mitigating the catastrophic forgetting problem in semi-supervised continual learning scenarios.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
