arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.42100cs.LG
收藏

探究可塑性损失对随机环境下强化学习的影响

Towards Understanding the Impact of Plasticity Loss on Reinforcement Learning in Stochastic Environments

University of Freiburg(弗莱堡大学);Karlsruhe Institute of Technology(卡尔斯鲁厄理工学院)

Philipp Bordne, André Biedenkapp

AI总结:

该研究针对随机环境下强化学习的可塑性损失问题,通过在带噪声的DeepMind Control任务上训练SAC智能体,揭示了可塑性保留方法的鲁棒性优势及随机性对可塑性的影响,明确了训练早期TD不稳定性为主要失效模式。

摘要:

深度强化学习(RL)是一种强大的序列决策框架,尽管近年来取得了诸多引人注目的成功,但深度RL的脆弱性是公认的。新兴研究将这种脆弱性归因于神经网络的可塑性损失,然而,可塑性损失迄今在随机环境中尚未得到充分研究。我们系统研究了这种相互作用,在三个DeepMind Control任务上训练SAC智能体,这些任务添加了可控水平的观测噪声和动作噪声。我们发现,保留可塑性的方法在两个较难的任务上提高了学习鲁棒性,其优势随观测噪声的增加而增大;在所有测试的观测噪声水平下,结合重置与层归一化的表现均不如单独使用重置。此外,随机性会在更困难的环境中加速可塑性损失,而在较简单的环境中,噪声可作为正则化项维持可塑性。最后,可塑性损失在训练早期最为显著;在最具挑战性的随机环境中,TD不稳定性而非渐进式可塑性损失成为主导失效模式。

英文摘要:

Deep reinforcement learning (RL) is a powerful framework for sequential decision-making. Despite many high-profile successes in recent years, deep RL is known to be brittle. An emerging line of research attributes this brittleness to a loss of plasticity in neural networks. However, plasticity loss has so far been understudied in stochastic environments. We study this interplay systematically, training SAC agents on three DeepMind Control tasks augmented with controlled levels of observation and action noise. We find that plasticity-preserving methods increase learning robustness on the two harder tasks, with their advantage growing as observation noise increases, while combining resets with layer normalization underperforms resets alone at every observation noise level tested. Further, stochasticity can accelerate the loss of plasticity in harder environments, whereas in easier settings noise can act as a regularizer that sustains plasticity. Lastly, plasticity loss is most pronounced in the early stages of training. In the most challenging stochastic settings, TD instability rather than gradual plasticity loss becomes the dominant failure mode.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑