面向可扩展深度强化学习的超球面归一化
Hyperspherical Normalization for Scalable Deep Reinforcement Learning
AI总结：
针对深度强化学习扩大模型规模易导致过拟合与优化不稳定的问题，提出SimbaV2架构，通过超球面归一化约束权重与特征范数增长，并采用带奖励缩放的分布式价值估计稳定梯度，在4个领域的57个连续控制任务上实现了最先进的性能。
摘要：
扩大模型规模和计算量在监督学习中带来了持续的性能提升。然而，这一经验往往不适用于强化学习（RL），因为在非平稳数据上训练模型容易导致过拟合和优化不稳定。为此，我们引入了SimbaV2，这是一种新颖的RL架构，旨在通过（i）利用超球面归一化来约束权重和特征范数的增长；以及（ii）使用带有奖励缩放的分布式价值估计，以在不同奖励幅度下保持稳定的梯度，从而稳定优化过程。以soft actor-critic作为基础算法，SimbaV2能够随着更大的模型和更多的计算量有效扩展，在4个领域的57个连续控制任务上实现了最先进的性能。代码可在https://dojeon-ai.github.io/SimbaV2获取。
英文摘要：
Scaling up the model size and computation has brought consistent performance improvements in supervised learning. However, this lesson often fails to apply to reinforcement learning (RL) because training the model on non-stationary data easily leads to overfitting and unstable optimization. In response, we introduce SimbaV2, a novel RL architecture designed to stabilize optimization by (i) constraining the growth of weight and feature norm by hyperspherical normalization; and (ii) using a distributional value estimation with reward scaling to maintain stable gradients under varying reward magnitudes. Using the soft actor-critic as a base algorithm, SimbaV2 scales up effectively with larger models and greater compute, achieving state-of-the-art performance on 57 continuous control tasks across 4 domains. The code is available at https://dojeon-ai.github.io/SimbaV2.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
