调整你的教师网络：改进无样本持续学习中的知识蒸馏
Adapt Your Teacher: Improving Knowledge Distillation for Exemplar-free Continual Learning
AI总结：
针对无样本类别增量学习中知识蒸馏因教师网络表征偏移导致性能下降的问题，提出同步更新教师与主模型的教师自适应方法，可无缝集成于现有基于KD的CIL方法，在多个基准上实现性能提升。
摘要：
在本研究中，我们探究了以知识蒸馏（KD，Knowledge Distillation）作为正则化策略的无样本类别增量学习（CIL，class incremental learning），旨在防止遗忘问题。基于KD的方法已在CIL中得到成功应用，但在无法获取先前任务训练数据样本的情况下，这类方法往往难以对模型实现有效正则化。我们的分析表明，这一问题源于教师网络在处理分布外数据时出现的显著表征偏移，这种偏移会导致KD损失分量产生较大误差，进而造成CIL模型的性能下降。受近期测试时自适应方法的启发，我们提出了教师自适应（TA，Teacher Adaptation）方法，该方法在增量训练过程中同步更新教师模型与主模型。我们的方法可与基于KD的CIL方法无缝集成，在多个无样本CIL基准测试中均能持续提升这类方法的性能。本方法的源代码可在https://github.com/fszatkowski/cl-teacher-adaptation获取。
英文摘要：
In this work, we investigate exemplar-free class incremental learning (CIL) with knowledge distillation (KD) as a regularization strategy, aiming to prevent forgetting. KD-based methods are successfully used in CIL, but they often struggle to regularize the model without access to exemplars of the training data from previous tasks. Our analysis reveals that this issue originates from substantial representation shifts in the teacher network when dealing with out-of-distribution data. This causes large errors in the KD loss component, leading to performance degradation in CIL models. Inspired by recent test-time adaptation methods, we introduce Teacher Adaptation (TA), a method that concurrently updates the teacher and the main models during incremental training. Our method seamlessly integrates with KD-based CIL approaches and allows for consistent enhancement of their performance across multiple exemplar-free CIL benchmarks. The source code for our method is available at https://github.com/fszatkowski/cl-teacher-adaptation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
