arXiv 2203.172692023-04-03cs.LGcs.AIcs.CV

深入研究无排练持续学习

A Closer Look at Rehearsal-Free Continual Learning

Georgia Institute of Technology（佐治亚理工学院）；Samsung Research America（三星美国研究院）

James Seale Smith, Junjiao Tian, Shaunak Halbe, Yen-Chang Hsu, Zsolt Kira

备注：: Accepted by the 2023 IEEE/CVF Conference on Computer Vision and Pattern (CVPR) Workshop on Continual Learning in Computer Vision (CLVision 2023)

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文研究无排练持续学习，通过结合知识蒸馏与参数正则化，发现L2参数正则化在ViT自注意力模块中优于EWC、特征蒸馏及提示方法。

摘要：

持续学习是一种机器学习模型从不断变化的训练数据中学习新概念的设定，同时避免对先前已见类别的知识退化，这些类别可能在训练数据中消失很长时间（这种现象被称为灾难性遗忘问题）。当前针对单一扩展任务（即类增量持续学习）的持续学习方法需要大量排练先前见过的数据以避免这种知识退化。遗憾的是，排练会带来内存成本，并且可能违反数据隐私。相反，我们探索以新方式结合知识蒸馏和参数正则化，以在不排练的情况下实现强大的持续学习性能。具体而言，我们深入研究了常见的持续学习技术：预测蒸馏、特征蒸馏、L2参数正则化和EWC参数正则化。我们首先反驳了参数正则化技术在单一扩展任务的无排练持续学习中失败的常见假设。接下来，我们探索如何在无排练持续学习中利用预训练模型的知识，发现原始的L2参数正则化优于EWC参数正则化和特征蒸馏。最后，我们探索了近期流行的ImageNet-R基准，并表明在ViT transformer的自注意力模块中实现的L2参数正则化优于近期流行的用于持续学习方法的提示技术。

英文摘要：

Continual learning is a setting where machine learning models learn novel concepts from continuously shifting training data, while simultaneously avoiding degradation of knowledge on previously seen classes which may disappear from the training data for extended periods of time (a phenomenon known as the catastrophic forgetting problem). Current approaches for continual learning of a single expanding task (aka class-incremental continual learning) require extensive rehearsal of previously seen data to avoid this degradation of knowledge. Unfortunately, rehearsal comes at a cost to memory, and it may also violate data-privacy. Instead, we explore combining knowledge distillation and parameter regularization in new ways to achieve strong continual learning performance without rehearsal. Specifically, we take a deep dive into common continual learning techniques: prediction distillation, feature distillation, L2 parameter regularization, and EWC parameter regularization. We first disprove the common assumption that parameter regularization techniques fail for rehearsal-free continual learning of a single, expanding task. Next, we explore how to leverage knowledge from a pre-trained model in rehearsal-free continual learning and find that vanilla L2 parameter regularization outperforms EWC parameter regularization and feature distillation. Finally, we explore the recently popular ImageNet-R benchmark, and show that L2 parameter regularization implemented in self-attention blocks of a ViT transformer outperforms recent popular prompting for continual learning methods.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。