Deep reinforcement learning (RL) is a powerful framework for sequential decision-making. Despite many high-profile successes in recent years, deep RL is known to be brittle. An emerging line of research attributes this brittleness to a loss of plasticity in neural networks. However, plasticity loss has so far been understudied in stochastic environments. We study this interplay systematically, training SAC agents on three DeepMind Control tasks augmented with controlled levels of observation and action noise. We find that plasticity-preserving methods increase learning robustness on the two harder tasks, with their advantage growing as observation noise increases, while combining resets with layer normalization underperforms resets alone at every observation noise level tested. Further, stochasticity can accelerate the loss of plasticity in harder environments, whereas in easier settings noise can act as a regularizer that sustains plasticity. Lastly, plasticity loss is most pronounced in the early stages of training. In the most challenging stochastic settings, TD instability rather than gradual plasticity loss becomes the dominant failure mode.
Stagnant Neuron: Towards Understanding the Plasticity Loss in Multi-Agent Reinforcement Learning Value Factorization Methods
Fujian Key Laboratory of Urban Intelligent Sensing and Computing, School of Informatics, Xiamen University (XMU)(福建省城市智能感知与计算重点实验室,厦门大学信息学院);Key Laboratory of Multimedia Trusted Perception and Efficient Computing, XMU(多媒体可信感知与高效计算重点实验室,厦门大学)
The Rank and Gradient Lost in Non-stationarity: Sample Weight Decay for Mitigating Plasticity Loss in Reinforcement Learning
School of Computer Software, Tianjin University(天津大学计算机软件学院);International Joint Institute of Tianjin University, Fuzhou(天津大学福州国际联合学院);The Hong Kong University of Science and Technology(香港科技大学)
Plasticity Loss in Deep Reinforcement Learning: A Survey
Faculty of Computer Science, University of Vienna(维也纳大学计算机科学系);Doctoral School Computer Science, University of Vienna(维也纳大学计算机科学博士学院);Independent Researcher(独立研究员);Microsoft AI(微软人工智能);Department of Machine Learning and Systems Biology, Max Planck Institute of Biochemistry(生物化学研究所机器学习与系统生物学部门)
Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stages
Tsinghua University(清华大学);The University of Sydney(悉尼大学);Washington University in St. Louis(圣路易斯华盛顿大学);JD Explore Academy(京东探索研究院);Nanyang Technological University(南洋理工大学)
Neuron-level Balance between Stability and Plasticity in Deep Reinforcement Learning
School of Software, Beihang University(北京航空航天大学软件学院);School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University(中山大学深圳校区网络空间科学与技术学院);Nanyang Technological University(南洋理工大学)
Embodied Synaptic Plasticity with Online Reinforcement learning
FZI Research Center for Information Technology(FZI信息技术研究中心);Graz University of Technology(格拉茨技术大学);Georg-August Universität(哥廷根乔治-奥古斯特大学);Technische Universität Dresden(德累斯顿工业大学)
Mastering Continual Reinforcement Learning through Fine-Grained Sparse Network Allocation and Dormant Neuron Exploration
IHPC, Agency for Science, Technology and Research (A*STAR)(新加坡科技研究局IHPC);College of Computing and Data Science, Nanyang Technological University (NTU)(南洋理工大学计算与数据科学学院)
Revisiting Clustering of Neural Bandits: Selective Reinitialization for Mitigating Loss of Plasticity
Gaoling School of Artificial Intelligence Renmin University of China Beijing China;Gaoling School of Artificial Intelligence Renmin University of China
Balancing Plasticity and Stability with Fast and Slow Successor Features
School of Computer Science, McGill University, Montr\'eal, Canada(谷歌DeepMind);Mila - Quebec Artificial Intelligence Institute, Montr\'eal, QC, Canada;CIFAR Learning in Machines;Google Deepmind;Montreal Neurological Institute of McGill University, Montr\'eal, Canada
School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);Sun Yat-sen University(中山大学);Lehigh University(里海大学);College of Computing and Data Science, NTU(南洋理工大学计算与数据科学学院)
Self-Composing Policies for Scalable Continual Reinforcement Learning
Department of Computer Science and Artificial Intelligence, University of the Basque Country UPV/EHU, Donostia-San Sebastian, Spain(计算机科学与人工智能系,巴斯克国家大学UPV/EHU,圣地亚哥-桑塞班西斯,西班牙);Basque Center for Applied Mathematics (BCAM), Bilbao, Spain(应用数学巴斯克中心(BCAM),毕尔巴鄂,西班牙)
SPHERE: Mitigating the Loss of Spectral Plasticity in Mixture-of-Experts for Deep Reinforcement Learning
State Key Lab of General AI, School of Intelligence Science and Technology, Peking University(人工智能国家重点实验室,智能科学与技术学院,北京大学);State Key Laboratory of General Artificial Intelligence, BIGAI(通用人工智能国家重点实验室,BIGAI);Institute for Artificial Intelligence, Peking University(人工智能研究院,北京大学)
When RL Fails after SFT: Rejuvenating Model Plasticity for Robust SFT-to-RL Handoff
Hong Kong University of Science and Technology(香港科技大学);Zhejiang University(浙江大学);State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA(多模态人工智能系统国家重点实验室,CASIA)