基于单侧超估计非对称损失近似的持续学习
Continual Learning by Asymmetric Loss Approximation with Single-Side Overestimation
摘要:
灾难性遗忘是训练深度神经网络的关键挑战。虽然持续学习已被调查作为一个对策的问题,它往往受到额外的网络组件的要求和有限的可扩展性,大量的任务。我们提出了一种新的方法来持续学习,通过近似一个真正的损失函数使用一个不对称的二次函数,其一侧高估。我们的算法是由经验观察,网络参数更新影响目标损失函数不对称的动机。在所提出的持续学习框架中,我们通过在训练新任务时适当高估其未观察侧来估计过去考虑的任务的非对称损失函数,同时导出可观察侧的精确模型参数。与现有的方法相比,我们的方法没有副作用,并且在几个具有挑战性的基准数据集上实现了最先进的精度,甚至接近上限性能。
英文摘要:
Catastrophic forgetting is a critical challenge in training deep neural networks. Although continual learning has been investigated as a countermeasure to the problem, it often suffers from the requirements of additional network components and the limited scalability to a large number of tasks. We propose a novel approach to continual learning by approximating a true loss function using an asymmetric quadratic function with one of its sides overestimated. Our algorithm is motivated by the empirical observation that the network parameter updates affect the target loss functions asymmetrically. In the proposed continual learning framework, we estimate an asymmetric loss function for the tasks considered in the past through a proper overestimation of its unobserved sides in training new tasks, while deriving the accurate model parameter for the observable sides. In contrast to existing approaches, our method is free from the side effects and achieves the state-of-the-art accuracy that is even close to the upper-bound performance on several challenging benchmark datasets.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。