SLCA: 面向预训练模型持续学习的带分类器对齐的慢学习器方法
SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Model
AI总结：
针对预训练模型持续学习中的渐进式过拟合问题，提出SLCA方法，通过选择性降低表示层学习率、建模类别分布并事后对齐分类层，在多个基准数据集上大幅超越现有最优方法。
摘要：
持续学习的目标是提升识别模型在学习依次到达的数据时的性能。尽管现有大多数研究都建立在从零开始学习的前提之上，但越来越多的工作致力于融合预训练的优势。然而，如何为每个增量任务自适应地利用预训练知识，同时保持其泛化能力，仍然是一个悬而未决的问题。在本研究中，我们对基于预训练模型的持续学习（CLPM）开展了全面分析，并将核心挑战归结为渐进式过拟合问题。我们观察到，有选择地降低学习率几乎可以在表示层解决这一问题，因此我们提出了一种简单但极其有效的方法，名为带分类器对齐的慢学习器（SLCA），该方法通过建模类别分布并以事后方式对齐分类层，进一步优化了分类层。在多种场景下，我们的方法为CLPM带来了显著提升（例如，在Split CIFAR-100、Split ImageNet-R、Split CUB-200和Split Cars-196上分别最高提升49.76%、50.05%、44.69%和40.16%），从而大幅超越了当前最优方法。基于这一强大的基线，我们深入分析了关键影响因素和有前景的研究方向，以推动后续研究。代码已公开于：https://github.com/GengDavid/SLCA。
英文摘要：
The goal of continual learning is to improve the performance of recognition models in learning sequentially arrived data. Although most existing works are established on the premise of learning from scratch, growing efforts have been devoted to incorporating the benefits of pre-training. However, how to adaptively exploit the pre-trained knowledge for each incremental task while maintaining its generalizability remains an open question. In this work, we present an extensive analysis for continual learning on a pre-trained model (CLPM), and attribute the key challenge to a progressive overfitting problem. Observing that selectively reducing the learning rate can almost resolve this issue in the representation layer, we propose a simple but extremely effective approach named Slow Learner with Classifier Alignment (SLCA), which further improves the classification layer by modeling the class-wise distributions and aligning the classification layers in a post-hoc fashion. Across a variety of scenarios, our proposal provides substantial improvements for CLPM (e.g., up to 49.76%, 50.05%, 44.69% and 40.16% on Split CIFAR-100, Split ImageNet-R, Split CUB-200 and Split Cars-196, respectively), and thus outperforms state-of-the-art approaches by a large margin. Based on such a strong baseline, critical factors and promising directions are analyzed in-depth to facilitate subsequent research. Code has been made available at: https://github.com/GengDavid/SLCA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
