通过动态等距保持持续学习中的可塑性
Preserving Plasticity in Continual Learning via Dynamical Isometry
AI总结：
本文通过动态等距机制保持深度神经网络在持续学习中的可塑性，提出等距正则化方法和AdamO优化器，在多个基准上匹配或超越现有方法。
摘要：
深度神经网络在非平稳条件下的持续训练通常会导致可塑性逐渐丧失，最终限制进一步学习。我们将可塑性与经验神经正切核联系起来，并确定动态等距（即逐层雅可比奇异值保持接近1的条件）是保持持续学习中可塑性的关键机制。我们重新审视一类几乎处处等距且同时保持通用Lipschitz函数逼近能力的网络，证明近动态等距与表达性非线性表示兼容。对于通用架构，我们提出一种高效的等距促进正则化方案，并识别出一种可以重新激活休眠ReLU单元的新机制。在此基础上，我们引入AdamO，一种Adam风格的自适应优化器，将等距正则化与梯度更新解耦，类似于AdamW。我们进一步通过动态等距的视角重新解释先前的可塑性保持方法，表明它们仅针对等距的部分度量。在旨在诱导可塑性损失的监督和强化学习持续学习基准上，我们的方法一致地匹配或超越现有方法。
英文摘要：
Continual training of deep neural networks under non-stationarity often leads to a progressive loss of plasticity, eventually limiting further learning. We relate plasticity to the empirical Neural Tangent Kernel, and identify dynamical isometry (the condition that layer-wise Jacobian singular values remain close to one) as a key mechanism for preserving plasticity in continual learning. We revisit a class of networks that are almost-everywhere isometric while remaining universal Lipschitz function approximators, demonstrating that near-dynamical isometry is compatible with expressive nonlinear representations. For general architectures, we propose an efficient isometry-promoting regularization scheme and identify a novel mechanism by which it can reactivate dormant ReLU units. Building on this, we introduce AdamO, an Adam-style adaptive optimizer that decouples isometry regularization from gradient updates, analogous to AdamW. We further reinterpret prior plasticity-preserving approaches through the lens of dynamical isometry, showing that they target only a partial measure of isometry. Across supervised and reinforcement-learning continual-learning benchmarks designed to induce plasticity loss, our methods consistently match or outperform existing approaches.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
