基于神经坍塌对比的内存高效持续学习
Memory-efficient Continual Learning with Neural Collapse Contrastive
AI总结：
针对持续学习中的灾难性遗忘与内存依赖问题，提出 FNC^2 平衡软硬关系，并结合 HSD 跨任务蒸馏知识，在无内存条件下仍可媲美基于回放的方法。
摘要：
对比学习显著提升了表征质量，增强了持续学习（CL）中跨任务的知识迁移。然而，灾难性遗忘仍是一项关键挑战，因为基于对比的方法主要关注样本之间会随数据分布变化而漂移的“软关系”或“软性”，从而导致跨任务的表征重叠。最近，新发现的 Neural Collapse（神经坍塌）现象通过关注样本与固定原型之间的“硬关系”或“硬性”，在 CL 中展现出前景。但是，该方法忽略了对捕获类内变化至关重要的“软性”，而且这种刚性关注还可能将旧类别表征拉向当前类别，加剧遗忘。基于这些洞见，我们提出 Focal Neural Collapse Contrastive（FNC^2），这是一种新颖的表征学习损失，可有效平衡软关系与硬关系。此外，我们引入 Hardness-Softness Distillation（HSD）损失，以在跨任务过程中逐步保留从这些关系中获得的知识。我们的方法优于最先进方法，尤其是在最小化内存依赖方面。值得注意的是，即使不使用内存，我们的方法也能媲美基于回放的方法，为数据隐私问题提供了有吸引力的解决方案。
英文摘要：
Contrastive learning has significantly improved representation quality, enhancing knowledge transfer across tasks in continual learning (CL). However, catastrophic forgetting remains a key challenge, as contrastive based methods primarily focus on "soft relationships" or "softness" between samples, which shift with changing data distributions and lead to representation overlap across tasks. Recently, the newly identified Neural Collapse phenomenon has shown promise in CL by focusing on "hard relationships" or "hardness" between samples and fixed prototypes. However, this approach overlooks "softness", crucial for capturing intra-class variability, and this rigid focus can also pull old class representations toward current ones, increasing forgetting. Building on these insights, we propose Focal Neural Collapse Contrastive (FNC^2), a novel representation learning loss that effectively balances both soft and hard relationships. Additionally, we introduce the Hardness-Softness Distillation (HSD) loss to progressively preserve the knowledge gained from these relationships across tasks. Our method outperforms state-of-the-art approaches, particularly in minimizing memory reliance. Remarkably, even without the use of memory, our approach rivals rehearsal-based methods, offering a compelling solution for data privacy concerns.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
