面向无监督持续学习的可塑性优化互补网络
Plasticity-Optimized Complementary Networks for Unsupervised Continual Learning
AI总结：
提出一种可塑性优化互补网络，通过训练无遗忘职责的专家网络并在适应-回顾阶段融合新旧知识，解决了无监督持续学习中的可塑性-稳定性权衡问题，在多任务设置下优于其他无样本方法。
摘要：
连续无监督表示学习（CURL）研究极大地受益于自监督学习（SSL）技术的进步。因此，现有的基于SSL的CURL方法能够在没有任何标签的情况下学习高质量的表示，但在多任务数据流上学习时性能会显著下降。我们假设这是由于为防止遗忘而施加的正则化损失导致的，从而产生了次优的可塑性-稳定性权衡：它们要么没有完全适应输入数据（低可塑性），要么在允许完全适应新的SSL pretext-task时发生显著遗忘（低稳定性）。在这项工作中，我们提出训练一个专家网络，使其免除保留先前知识的职责，并能够专注于在新任务上实现最优表现（优化可塑性）。在第二阶段，我们在适应-回顾阶段将这一新知识与先前的网络相结合，以避免遗忘，并利用旧网络的知识初始化一个新的专家。我们进行了多项实验，表明我们提出的方法在少任务和多任务分割设置中优于其他无样本CURL方法。此外，我们展示了如何将我们的方法适应于半监督持续学习（Semi-SCL），并表明我们超越了其他无样本Semi-SCL方法的准确性，并达到了一些使用样本的方法的结果。
英文摘要：
Continuous unsupervised representation learning (CURL) research has greatly benefited from improvements in self-supervised learning (SSL) techniques. As a result, existing CURL methods using SSL can learn high-quality representations without any labels, but with a notable performance drop when learning on a many-tasks data stream. We hypothesize that this is caused by the regularization losses that are imposed to prevent forgetting, leading to a suboptimal plasticity-stability trade-off: they either do not adapt fully to the incoming data (low plasticity), or incur significant forgetting when allowed to fully adapt to a new SSL pretext-task (low stability). In this work, we propose to train an expert network that is relieved of the duty of keeping the previous knowledge and can focus on performing optimally on the new tasks (optimizing plasticity). In the second phase, we combine this new knowledge with the previous network in an adaptation-retrospection phase to avoid forgetting and initialize a new expert with the knowledge of the old network. We perform several experiments showing that our proposed approach outperforms other CURL exemplar-free methods in few- and many-task split settings. Furthermore, we show how to adapt our approach to semi-supervised continual learning (Semi-SCL) and show that we surpass the accuracy of other exemplar-free Semi-SCL methods and reach the results of some others that use exemplars.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
