arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.36309cs.LGcs.RO
收藏

桥接在线策略强化学习中的探索与可塑性丧失

Bridging Exploration and Loss of Plasticity in On-Policy Reinforcement Learning

Politecnico di Torino(都灵理工大学);Istituto Italiano di Tecnologia(意大利技术研究院)

Andrea Vasco Grieco, Evelyn D'Elia, Edoardo Caldarelli, Raffaello Camoriano

AI总结:

该研究针对在线策略强化学习中的可塑性丧失(LoP)问题,以近端策略优化(PPO)为对象,分析了策略标准差σ对LoP的影响,结合理论与实验验证,还在7自由度机械臂上完成了真实策略部署。

摘要:

深度强化学习(RL)算法会丧失从新数据中自适应学习的能力,该现象被称为可塑性丧失(Loss of Plasticity, LoP)。尽管已有多种指标被证实与LoP相关联,但其产生的因果机制和几何条件尚未明确。我们分析了采用各向同性高斯策略的近端策略优化(Proximal Policy Optimization, PPO)中的LoP,PPO是RL中许多连续动作空间的最先进方案。我们报告了多个表征策略标准差σ对LoP影响的指标,并对广泛采用的LoP缓解技术的有效性给出了相关解释。我们观察到σ对LoP程度有显著影响,并辅以理论和实验支撑论据。我们的实验验证包括针对标准基线的LoP基准评估,以及在7自由度(7-DoF)机械臂上的真实世界策略部署。

英文摘要:

Deep reinforcement learning (RL) algorithms have been observed to lose the ability to adaptively learn from new data, a phenomenon termed Loss of Plasticity (LoP). While many metrics have been shown to correlate with it, the causal mechanisms and the geometric conditions from which it can arise have not been clearly identified. We analyze LoP in Proximal Policy Optimization with isotropic Gaussian policies, the state-of-the-art formulation for many continuous action spaces in RL. We report multiple metrics which characterize the effect that the policy standard deviation $σ$ has on LoP, and we also suggest related explanations for the effectiveness of widely adopted LoP mitigation techniques. We observe a significant effect of $σ$ on the extent of LoP, and accompany this with both theoretical and experimental supporting arguments. Our experimental validation consists of LoP benchmark evaluations against standard baselines as well as real-world policy deployment on a 7-DoF robotic arm.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑