随机表示优于在线持续学习到的表示
Random Representations Outperform Online Continually Learned Representations
AI总结：
本文提出RanDumb方法，用固定随机变换加线性分类器证明简单随机表示在在线持续学习中优于持续学习到的表示，并挑战了现有表示学习假设。
摘要：
持续学习主要关注灾难性遗忘问题以及相关的稳定性-可塑性权衡。然而，对于持续学习到的表示的有效性却很少受到关注，因为在学习过程中表示是与分类器一起被学习的。我们的主要贡献是通过实验证明，现有的在线持续训练的深度网络产生的表示不如简单的预定义随机变换。我们的方法使用固定的随机变换对原始像素进行投影，在任何数据被看到之前近似初始化一个RBF核。然后，我们在其上训练一个简单的线性分类器，不存储任何样本，在在线持续学习设置中一次处理一个样本。这种称为RanDumb的方法在所有标准在线持续学习基准上显著优于最先进的持续学习表示。我们的研究揭示了表示学习的显著局限性，特别是在低样本和在线持续学习场景中。将我们的研究扩展到使用预训练模型的流行的无样本场景中，我们发现仅在预训练表示之上训练一个线性分类器就超过了大多数持续微调和提示调优策略。总体而言，我们的研究挑战了关于在线持续学习中有效表示学习的普遍假设。我们的代码可在以下地址获取：//github.com/drimpossible/RanDumb。
英文摘要：
Continual learning has primarily focused on the issue of catastrophic forgetting and the associated stability-plasticity tradeoffs. However, little attention has been paid to the efficacy of continually learned representations, as representations are learned alongside classifiers throughout the learning process. Our primary contribution is empirically demonstrating that existing online continually trained deep networks produce inferior representations compared to a simple pre-defined random transforms. Our approach projects raw pixels using a fixed random transform, approximating an RBF-Kernel initialized before any data is seen. We then train a simple linear classifier on top without storing any exemplars, processing one sample at a time in an online continual learning setting. This method, called RanDumb, significantly outperforms state-of-the-art continually learned representations across all standard online continual learning benchmarks. Our study reveals the significant limitations of representation learning, particularly in low-exemplar and online continual learning scenarios. Extending our investigation to popular exemplar-free scenarios with pretrained models, we find that training only a linear classifier on top of pretrained representations surpasses most continual fine-tuning and prompt-tuning strategies. Overall, our investigation challenges the prevailing assumptions about effective representation learning in online continual learning. Our code is available at://github.com/drimpossible/RanDumb.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
