探索持续学习中的稳定性差距：分类头的作用
Exploring the Stability Gap in Continual Learning: The Role of the Classification Head
AI总结：
针对持续学习中的稳定性差距（模型先快速下降后部分恢复性能）问题，研究聚焦分类头作用，引入最近均值分类器（NMC），提升了多种基准的训练稳定性与最终性能，揭示线性头是该现象主因。
摘要：
持续学习（CL）已成为机器学习的关键领域，它使神经网络能够从不断演变的数据分布中学习，同时减轻灾难性遗忘。然而，近期研究发现了稳定性差距——一种模型在训练期间先在已学任务上性能下降，随后部分恢复的现象。这种学习动态与持续学习中稳定性的直观理解相矛盾，人们本期望性能会逐渐下降，而非先快速降低再部分恢复。为更好理解并缓解稳定性差距，我们在神经网络架构的不同层级对其展开研究，尤其聚焦分类头的作用。我们引入最近均值分类器（NMC）作为工具，以归因骨干网络和分类头对稳定性差距的影响。实验表明，NMC不仅提升了最终性能，还显著增强了多种持续学习基准（包括CIFAR100、ImageNet100、CUB-200和FGVC Aircrafts）上的训练稳定性。此外，我们发现NMC还减少了任务近期偏差。我们的分析为稳定性差距提供了新见解，并表明该现象的主要成因是线性头，而非表征学习不足。
英文摘要：
Continual learning (CL) has emerged as a critical area in machine learning, enabling neural networks to learn from evolving data distributions while mitigating catastrophic forgetting. However, recent research has identified the stability gap -- a phenomenon where models initially lose performance on previously learned tasks before partially recovering during training. Such learning dynamics are contradictory to the intuitive understanding of stability in continual learning where one would expect the performance to degrade gradually instead of rapidly decreasing and then partially recovering later. To better understand and alleviate the stability gap, we investigate it at different levels of the neural network architecture, particularly focusing on the role of the classification head. We introduce the nearest-mean classifier (NMC) as a tool to attribute the influence of the backbone and the classification head on the stability gap. Our experiments demonstrate that NMC not only improves final performance, but also significantly enhances training stability across various continual learning benchmarks, including CIFAR100, ImageNet100, CUB-200, and FGVC Aircrafts. Moreover, we find that NMC also reduces task-recency bias. Our analysis provides new insights into the stability gap and suggests that the primary contributor to this phenomenon is the linear head, rather than the insufficient representation learning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
