克服持续学习中归一化统计的近期偏差：平衡与自适应
Overcoming Recency Bias of Normalization Statistics in Continual Learning: Balance and Adaptation
AI总结：
针对持续学习中批量归一化统计的近期偏差问题，提出自适应平衡BN（AdaB$^2$N）方法，结合贝叶斯策略和修正动量，在多个基准上显著提升性能。
摘要：
持续学习涉及学习一系列任务并适当平衡它们的知识。由于对旧训练样本的访问受限，当前深度神经网络领域的大部分工作都集中在基于梯度的优化中克服旧任务的灾难性遗忘上。然而，归一化层提供了一个例外，因为它们由当前观察到的训练样本的梯度和统计量相互依赖地更新，这需要专门的策略来减轻近期偏差。在这项工作中，我们聚焦于最流行的批量归一化（BN），并对其在持续学习中的次优性进行了深入的理论分析。我们的分析揭示了BN统计量在增量任务中平衡与自适应之间的困境，这可能影响训练稳定性和泛化能力。针对这些特定挑战，我们提出了自适应平衡BN（AdaB$^2$N），它适当地结合了基于贝叶斯的策略来调整任务贡献，以及一个修正的动量来平衡BN统计量，分别对应训练和测试阶段。通过以持续学习的方式实现BN，我们的方法在广泛的基准上取得了显著的性能提升，特别是在具有挑战性但现实场景的在线设置中（例如，在Split CIFAR-10、Split CIFAR-100和Split Mini-ImageNet上分别提升高达7.68%、6.86%和4.26%）。我们的代码可在https://github.com/lvyilin/AdaB2N获取。
英文摘要：
Continual learning entails learning a sequence of tasks and balancing their knowledge appropriately. With limited access to old training samples, much of the current work in deep neural networks has focused on overcoming catastrophic forgetting of old tasks in gradient-based optimization. However, the normalization layers provide an exception, as they are updated interdependently by the gradient and statistics of currently observed training samples, which require specialized strategies to mitigate recency bias. In this work, we focus on the most popular Batch Normalization (BN) and provide an in-depth theoretical analysis of its sub-optimality in continual learning. Our analysis demonstrates the dilemma between balance and adaptation of BN statistics for incremental tasks, which potentially affects training stability and generalization. Targeting on these particular challenges, we propose Adaptive Balance of BN (AdaB$^2$N), which incorporates appropriately a Bayesian-based strategy to adapt task-wise contributions and a modified momentum to balance BN statistics, corresponding to the training and testing stages. By implementing BN in a continual learning fashion, our approach achieves significant performance gains across a wide range of benchmarks, particularly for the challenging yet realistic online scenarios (e.g., up to 7.68%, 6.86% and 4.26% on Split CIFAR-10, Split CIFAR-100 and Split Mini-ImageNet, respectively). Our code is available at https://github.com/lvyilin/AdaB2N.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
