激活函数设计维持持续学习中的可塑性
Activation Function Design Sustains Plasticity in Continual Learning
AI总结：
本文探讨了激活函数在持续学习中的作用，提出两种新的非线性函数以缓解可塑性损失，通过监督学习和强化学习实验验证了其有效性。
摘要：
在独立同分布训练中，激活函数已被广泛测试，其差异通常在模型规模和优化调整后缩小。然而在持续学习中，情况不同：除了灾难性遗忘外，模型可能逐渐失去适应能力（称为可塑性丧失），而非线性在这一失败模式中的作用仍被忽视。我们证明激活函数的选择是缓解可塑性损失的主要因素。基于负分支形状和饱和行为的层面分析，我们引入了两种即插式非线性函数（Smooth-Leaky和Randomized Smooth-Leaky），并在两个互补的设置中评估它们：（i）监督类增量基准和（ii）具有非平稳MuJoCo环境的强化学习，旨在诱导受控的分布和动态变化。我们还提供了一种简单的压力协议和诊断，将激活函数的形状与变化下的适应性联系起来。结论是明确的：经过深思熟虑的激活函数设计提供了一种轻量、通用的方法，在持续学习中无需额外容量或任务特定调整即可维持可塑性。
英文摘要：
In independent, identically distributed (i.i.d.) training regimes, activation functions have been benchmarked extensively, and their differences often shrink once model size and optimization are tuned. In continual learning, however, the picture is different: beyond catastrophic forgetting, models can progressively lose the ability to adapt (referred to as loss of plasticity) and the role of the non-linearity in this failure mode remains underexplored. We show that activation choice is a primary, architecture-agnostic lever for mitigating plasticity loss. Building on a property-level analysis of negative-branch shape and saturation behavior, we introduce two drop-in nonlinearities (Smooth-Leaky and Randomized Smooth-Leaky) and evaluate them in two complementary settings: (i) supervised class-incremental benchmarks and (ii) reinforcement learning with non-stationary MuJoCo environments designed to induce controlled distribution and dynamics shifts. We also provide a simple stress protocol and diagnostics that link the shape of the activation to the adaptation under change. The takeaway is straightforward: thoughtful activation design offers a lightweight, domain-general way to sustain plasticity in continual learning without extra capacity or task-specific tuning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
