结合重要性采样与原型-样本关系蒸馏的对比持续学习
Contrastive Continual Learning with Importance Sampling and Prototype-Instance Relation Distillation
AI总结：
针对传统持续学习的灾难性遗忘问题，提出结合重要性采样的对比持续学习方法CCLIS，含RBS策略与PRD损失，在标准基准上知识保留效果优于现有基线，有效缓解在线场景遗忘。
摘要：
近年来，由于对比学习方法能生成高质量表征，基于回放的对比持续学习被提出，用于探索如何持续学习可迁移的表征嵌入，以避免传统持续学习场景中的灾难性遗忘问题。基于该框架，我们提出了基于重要性采样的对比持续学习方法（Contrastive Continual Learning via Importance Sampling, CCLIS），通过一种新的回放缓冲区选择（Replay Buffer Selection, RBS）策略恢复过往数据分布来保留知识，该策略最小化估计方差，以保存难负样本用于高质量表征学习。此外，我们提出了原型-样本关系蒸馏（Prototype-instance Relation Distillation, PRD）损失，这是一种通过自蒸馏过程维持原型与样本表征之间关系的技术。在标准持续学习基准上的实验表明，我们的方法在知识保留方面显著优于现有基线，从而有效缓解了在线场景下的灾难性遗忘。代码可在https://github.com/lijy373/CCLIS获取。
英文摘要：
Recently, because of the high-quality representations of contrastive learning methods, rehearsal-based contrastive continual learning has been proposed to explore how to continually learn transferable representation embeddings to avoid the catastrophic forgetting issue in traditional continual settings. Based on this framework, we propose Contrastive Continual Learning via Importance Sampling (CCLIS) to preserve knowledge by recovering previous data distributions with a new strategy for Replay Buffer Selection (RBS), which minimize estimated variance to save hard negative samples for representation learning with high quality. Furthermore, we present the Prototype-instance Relation Distillation (PRD) loss, a technique designed to maintain the relationship between prototypes and sample representations using a self-distillation process. Experiments on standard continual learning benchmarks reveal that our method notably outperforms existing baselines in terms of knowledge preservation and thereby effectively counteracts catastrophic forgetting in online contexts. The code is available at https://github.com/lijy373/CCLIS.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
