决策边界感知的知识巩固生成更好的实例增量学习器
Decision Boundary-aware Knowledge Consolidation Generates Better Instance-Incremental Learner
AI总结：
本文提出一种决策边界感知的知识巩固蒸馏方法，通过在保留旧边界的同时适度扩展至失败案例，解决了实例增量学习中灾难性遗忘与概念漂移问题，并发现教师模型是更好的增量学习器。
摘要：
实例增量学习（IIL）专注于对同类数据进行持续学习。与类增量学习（CIL）相比，IIL很少被研究，因为它较少受到灾难性遗忘（CF）的影响。然而，除了保留知识外，在类别空间总是预先定义的真实世界部署场景中，利用潜在不可用的先前数据，进行持续且低成本的模型提升是更关键的需求。因此，我们首先定义了一个新的更实用的IIL设置，即在仅有新观测数据的情况下，除了抵抗CF外，还要提升模型性能。新IIL设置必须解决两个问题：1）因无法访问旧数据而导致的严重灾难性遗忘，以及2）因概念漂移而将现有决策边界扩展到新观测数据。为解决这些问题，我们的核心见解是在保留旧边界的同时，适度地将决策边界扩展到失败案例。因此，我们提出了一种新颖的决策边界感知蒸馏方法，通过将知识巩固给教师模型来缓解学生学习新知识的难度。我们还在现有数据集Cifar-100和ImageNet上建立了基准。值得注意的是，大量实验表明，教师模型可以成为比学生模型更好的增量学习器，这推翻了以往基于知识蒸馏的方法中学生模型作为主导角色的设定。
英文摘要：
Instance-incremental learning (IIL) focuses on learning continually with data of the same classes. Compared to class-incremental learning (CIL), the IIL is seldom explored because IIL suffers less from catastrophic forgetting (CF). However, besides retaining knowledge, in real-world deployment scenarios where the class space is always predefined, continual and cost-effective model promotion with the potential unavailability of previous data is a more essential demand. Therefore, we first define a new and more practical IIL setting as promoting the model's performance besides resisting CF with only new observations. Two issues have to be tackled in the new IIL setting: 1) the notorious catastrophic forgetting because of no access to old data, and 2) broadening the existing decision boundary to new observations because of concept drift. To tackle these problems, our key insight is to moderately broaden the decision boundary to fail cases while retain old boundary. Hence, we propose a novel decision boundary-aware distillation method with consolidating knowledge to teacher to ease the student learning new knowledge. We also establish the benchmarks on existing datasets Cifar-100 and ImageNet. Notably, extensive experiments demonstrate that the teacher model can be a better incremental learner than the student model, which overturns previous knowledge distillation-based methods treating student as the main role.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
