arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2112.04731cs.CVcs.LG
收藏

模仿 Oracle:一种用于类别增量学习的初始阶段去相关方法

Mimicking the Oracle: An Initial Phase Decorrelation Approach for Class Incremental Learning

National University of Singapore(新加坡国立大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);ByteDance Inc.(字节跳动公司);University of Oxford(牛津大学)

Yujun Shi, Kuangqi Zhou, Jian Liang, Zihang Jiang, Jiashi Feng, Philip Torr, Song Bai, Vincent Y. F. Tan

AI总结:

本文针对类别增量学习初始阶段表示过于狭长的问题,提出 Class-wise Decorrelation(CwD)正则化方法,使各类表示均匀分散以模仿全类别联合训练的 oracle 模型,并在多个基准上将现有方法稳定提升约 1%–3%。

摘要:

类别增量学习(Class Incremental Learning, CIL)旨在以逐阶段的方式学习多类分类器,其中每个阶段仅提供类别子集的数据。以往工作主要关注缓解初始阶段之后各阶段的遗忘问题。然而,我们发现改进 CIL 的初始阶段也是一个很有前景的方向。具体而言,我们通过实验表明,在初始阶段直接鼓励 CIL 学习器输出与在所有类别上联合训练的模型相似的表示,可以大幅提升 CIL 性能。受此启发,我们研究了朴素训练的初始阶段模型与 oracle 模型之间的差异。具体来说,由于这两个模型之间的一个主要差异在于训练类别的数量,我们考察了这种差异如何影响模型表示。我们发现,当训练类别较少时,每个类别的数据表示位于一个狭长区域;而当训练类别较多时,每个类别的表示会更均匀地分散。受此观察启发,我们提出了类别去相关(Class-wise Decorrelation, CwD),它有效正则化每个类别的表示,使其更均匀地分散,从而模仿在所有类别上联合训练的模型(即 oracle 模型)。我们的 CwD 实现简单,且易于插入现有方法。在多个基准数据集上的大量实验表明,CwD 持续且显著地将现有最先进方法的性能提高约 1% 至 3%。代码将会发布。

英文摘要:

Class Incremental Learning (CIL) aims at learning a multi-class classifier in a phase-by-phase manner, in which only data of a subset of the classes are provided at each phase. Previous works mainly focus on mitigating forgetting in phases after the initial one. However, we find that improving CIL at its initial phase is also a promising direction. Specifically, we experimentally show that directly encouraging CIL Learner at the initial phase to output similar representations as the model jointly trained on all classes can greatly boost the CIL performance. Motivated by this, we study the difference between a naïvely-trained initial-phase model and the oracle model. Specifically, since one major difference between these two models is the number of training classes, we investigate how such difference affects the model representations. We find that, with fewer training classes, the data representations of each class lie in a long and narrow region; with more training classes, the representations of each class scatter more uniformly. Inspired by this observation, we propose Class-wise Decorrelation (CwD) that effectively regularizes representations of each class to scatter more uniformly, thus mimicking the model jointly trained with all classes (i.e., the oracle model). Our CwD is simple to implement and easy to plug into existing methods. Extensive experiments on various benchmark datasets show that CwD consistently and significantly improves the performance of existing state-of-the-art methods by around 1\% to 3\%. Code will be released.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑