深度强化学习(RL)算法会丧失从新数据中自适应学习的能力,该现象被称为可塑性丧失(Loss of Plasticity, LoP)。尽管已有多种指标被证实与LoP相关联,但其产生的因果机制和几何条件尚未明确。我们分析了采用各向同性高斯策略的近端策略优化(Proximal Policy Optimization, PPO)中的LoP,PPO是RL中许多连续动作空间的最先进方案。我们报告了多个表征策略标准差σ对LoP影响的指标,并对广泛采用的LoP缓解技术的有效性给出了相关解释。我们观察到σ对LoP程度有显著影响,并辅以理论和实验支撑论据。我们的实验验证包括针对标准基线的LoP基准评估,以及在7自由度(7-DoF)机械臂上的真实世界策略部署。
英文摘要:
Deep reinforcement learning (RL) algorithms have been observed to lose the ability to adaptively learn from new data, a phenomenon termed Loss of Plasticity (LoP). While many metrics have been shown to correlate with it, the causal mechanisms and the geometric conditions from which it can arise have not been clearly identified. We analyze LoP in Proximal Policy Optimization with isotropic Gaussian policies, the state-of-the-art formulation for many continuous action spaces in RL. We report multiple metrics which characterize the effect that the policy standard deviation $σ$ has on LoP, and we also suggest related explanations for the effectiveness of widely adopted LoP mitigation techniques. We observe a significant effect of $σ$ on the extent of LoP, and accompany this with both theoretical and experimental supporting arguments. Our experimental validation consists of LoP benchmark evaluations against standard baselines as well as real-world policy deployment on a 7-DoF robotic arm.
Plasticity Loss in Deep Reinforcement Learning: A Survey
Faculty of Computer Science, University of Vienna(维也纳大学计算机科学系);Doctoral School Computer Science, University of Vienna(维也纳大学计算机科学博士学院);Independent Researcher(独立研究员);Microsoft AI(微软人工智能);Department of Machine Learning and Systems Biology, Max Planck Institute of Biochemistry(生物化学研究所机器学习与系统生物学部门)
Mastering Continual Reinforcement Learning through Fine-Grained Sparse Network Allocation and Dormant Neuron Exploration
IHPC, Agency for Science, Technology and Research (A*STAR)(新加坡科技研究局IHPC);College of Computing and Data Science, Nanyang Technological University (NTU)(南洋理工大学计算与数据科学学院)
DEP-RL: Embodied Exploration for Reinforcement Learning in Overactuated and Musculoskeletal Systems
Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所);Hertie-Institute for Clinical Brain Research(赫蒂临床脑研究所);Institute for Modelling and Simulation of Biomechanical Systems(生物力学系统建模与仿真研究所)
Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stages
Tsinghua University(清华大学);The University of Sydney(悉尼大学);Washington University in St. Louis(圣路易斯华盛顿大学);JD Explore Academy(京东探索研究院);Nanyang Technological University(南洋理工大学)
Embodied Synaptic Plasticity with Online Reinforcement learning
FZI Research Center for Information Technology(FZI信息技术研究中心);Graz University of Technology(格拉茨技术大学);Georg-August Universität(哥廷根乔治-奥古斯特大学);Technische Universität Dresden(德累斯顿工业大学)
Bridging SFT and RL: Dynamic Policy Optimization for Robust Reasoning
Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Intelligent System Department, Zhongxing Telecom Equipment (ZTE)(中兴通讯股份有限公司智能系统部);Institute of Advanced Interdisciplinary Studies, Chongqing University of Posts and Telecommunications(重庆邮电大学前沿交叉研究院)
PriorZero: Bridging Language Priors and World Models for Decision Making
University of Science and Technology of China(中国科学技术大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Nanjing University of Aeronautics and Astronautics(南京航空航天大学);The Chinese University of Hong Kong MMLab(香港中文大学MMLab)
Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Salesforce AI Research(Salesforce AI研究院);The Hong Kong University of Science and Technology(香港科技大学)