编排潜在专长：通过多级监督与逆向自蒸馏推进在线持续学习
Orchestrate Latent Expertise: Advancing Online Continual Learning with Multi-Level Supervision and Reverse Self-Distillation
AI总结：
针对在线持续学习中回放缓冲区导致的过拟合与欠拟合困境，提出多级在线顺序专家MOSE，通过多级监督与逆向自蒸馏提升新样本学习与旧知识保留能力。
摘要：
为了适应现实世界动态变化，人工智能系统需要以在线方式处理顺序到达的内容。常规持续学习（CL）试图通过对每个任务进行离线训练来解决灾难性遗忘问题，而在线持续学习（OCL）则是一种更具挑战性但也更贴近现实的设置，它在单遍数据流中执行持续学习。当前的OCL方法主要依赖对旧训练样本的记忆回放。然而，从CL到OCL存在一个显著差距，该差距源于与使用排练缓冲区相关的额外过拟合-欠拟合困境：对新训练样本学习不足（欠拟合）以及对少数旧训练样本的重复学习（过拟合）。为此，我们提出了一种新方法，即多级在线顺序专家（MOSE），它将模型培养为堆叠的子专家，融合多级监督与逆向自蒸馏。跨多个阶段的监督信号有助于新任务的适当收敛，同时通过知识蒸馏汇集各专家的不同优势，缓解旧任务的性能下降。MOSE在多级专家学习新样本和保留过去知识方面展现出显著效果，从而在OCL性能上大幅超越最先进的基线方法（例如，在Split CIFAR-100上提升高达7.3%，在Split Tiny-ImageNet上提升6.1%）。
英文摘要：
To accommodate real-world dynamics, artificial intelligence systems need to cope with sequentially arriving content in an online manner. Beyond regular Continual Learning (CL) attempting to address catastrophic forgetting with offline training of each task, Online Continual Learning (OCL) is a more challenging yet realistic setting that performs CL in a one-pass data stream. Current OCL methods primarily rely on memory replay of old training samples. However, a notable gap from CL to OCL stems from the additional overfitting-underfitting dilemma associated with the use of rehearsal buffers: the inadequate learning of new training samples (underfitting) and the repeated learning of a few old training samples (overfitting). To this end, we introduce a novel approach, Multi-level Online Sequential Experts (MOSE), which cultivates the model as stacked sub-experts, integrating multi-level supervision and reverse self-distillation. Supervision signals across multiple stages facilitate appropriate convergence of the new task while gathering various strengths from experts by knowledge distillation mitigates the performance decline of old tasks. MOSE demonstrates remarkable efficacy in learning new samples and preserving past knowledge through multi-level experts, thereby significantly advancing OCL performance over state-of-the-art baselines (e.g., up to 7.3% on Split CIFAR-100 and 6.1% on Split Tiny-ImageNet).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
