面向持续学习可塑性的自归一化重置
Self-Normalized Resets for Plasticity in Continual Learning
AI总结：
针对持续学习中的可塑性损失，本文提出自适应重置非活跃神经元的 SNR 算法，并通过实验与理论证明其性能、鲁棒性及学习单个 ReLU 时的优势。
摘要：
可塑性损失是一个日益重要的现象，指的是经验观察到的情况：随着神经网络在一系列不断变化的任务上持续训练，其适应新任务的能力会随时间下降。我们提出自归一化重置（Self-Normalized Resets，SNR），这是一种简单的自适应算法，当证据表明某个神经元的放电率实际上已降至零时，通过重置该神经元的权重来缓解可塑性损失。在一系列持续学习问题和网络架构中，我们证明 SNR 相比竞争算法始终取得更优性能。我们还证明 SNR 对其唯一超参数（即拒绝百分位阈值）具有鲁棒性，而竞争算法则表现出显著的敏感性。SNR 基于阈值的重置机制源自我们推导的一个简单假设检验。从该假设检验的视角看，相互竞争的重置方案在正确检测非活跃神经元时会产生次优错误率，这或许可以解释我们的实验观察。我们还对学习单个 ReLU 这一问题的优化景观进行了理论研究。我们表明，即使在对抗性初始化情况下，理想化版本的 SNR 也能学习目标 ReLU，而基于正则化的方法可能无法完成学习。
英文摘要：
Plasticity Loss is an increasingly important phenomenon that refers to the empirical observation that as a neural network is continually trained on a sequence of changing tasks, its ability to adapt to a new task diminishes over time. We introduce Self-Normalized Resets (SNR), a simple adaptive algorithm that mitigates plasticity loss by resetting a neuron's weights when evidence suggests its firing rate has effectively dropped to zero. Across a battery of continual learning problems and network architectures, we demonstrate that SNR consistently attains superior performance compared to its competitor algorithms. We also demonstrate that SNR is robust to its sole hyperparameter, its rejection percentile threshold, while competitor algorithms show significant sensitivity. SNR's threshold-based reset mechanism is motivated by a simple hypothesis test that we derive. Seen through the lens of this hypothesis test, competing reset proposals yield suboptimal error rates in correctly detecting inactive neurons, potentially explaining our experimental observations. We also conduct a theoretical investigation of the optimization landscape for the problem of learning a single ReLU. We show that even when initialized adversarially, an idealized version of SNR learns the target ReLU, while regularization based approaches can fail to learn.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
