强化学习中的归一化与有效学习率
Normalization and effective learning rates in reinforcement learning
AI总结：
本文针对归一化导致有效学习率随参数范数增长而隐式衰减的问题，提出 NaP 重参数化方法，使学习率显式恒定并提升强化学习与持续学习的非平稳鲁棒性。
摘要：
归一化层最近在深度强化学习和持续学习文献中经历了复兴，多项工作强调了其多样优势，例如改善损失景观条件数以及抑制高估偏差。然而，归一化带来了一个微妙但重要的副作用：网络参数范数增长与有效学习率衰减之间存在等价关系。这在持续学习场景中会成为问题，因为由此产生的有效学习率调度可能相对于学习问题的时间尺度过快地衰减至接近零。我们建议通过一种称为 Normalize-and-Project（NaP，归一化并投影）的简单重参数化，使学习率调度显式化；该方法将插入归一化层与权重投影结合，确保有效学习率在整个训练过程中保持恒定。这一技术不仅成为更深入理解深度强化学习中学习率调度的强大分析工具，也能在合成可塑性损失基准以及 Arcade Learning Environment 的单任务和序列变体中提升对非平稳性的鲁棒性。我们还表明，该方法可以轻松应用于 ResNets 和 transformers 等流行架构，同时在常见的平稳基准上恢复、并在某些情况下甚至略微提升基础模型的性能。
英文摘要：
Normalization layers have recently experienced a renaissance in the deep reinforcement learning and continual learning literature, with several works highlighting diverse benefits such as improving loss landscape conditioning and combatting overestimation bias. However, normalization brings with it a subtle but important side effect: an equivalence between growth in the norm of the network parameters and decay in the effective learning rate. This becomes problematic in continual learning settings, where the resulting effective learning rate schedule may decay to near zero too quickly relative to the timescale of the learning problem. We propose to make the learning rate schedule explicit with a simple re-parameterization which we call Normalize-and-Project (NaP), which couples the insertion of normalization layers with weight projection, ensuring that the effective learning rate remains constant throughout training. This technique reveals itself as a powerful analytical tool to better understand learning rate schedules in deep reinforcement learning, and as a means of improving robustness to nonstationarity in synthetic plasticity loss benchmarks along with both the single-task and sequential variants of the Arcade Learning Environment. We also show that our approach can be easily applied to popular architectures such as ResNets and transformers while recovering and in some cases even slightly improving the performance of the base model in common stationary benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
