arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2202.07152cs.ROcs.LG
收藏

L2C2:面向稳定与平滑强化学习的局部 Lipschitz 连续约束

L2C2: Locally Lipschitz Continuous Constraint towards Stable and Smooth Reinforcement Learning

Nara Institute of Science and Technology(奈良先端科学技术大学院大学)

Taisuke Kobayashi

AI总结:

本文提出L2C2正则化技术,通过基于状态转移设计时空局部紧致空间施加局部Lipschitz连续约束,使强化学习的策略与价值函数实现适度平滑,在不损失表达能力的前提下提升稳定性、抗噪性及任务性能。

摘要:

本文提出一种用于强化学习(RL)的新正则化技术,旨在使策略和价值函数平滑且稳定。RL 以学习过程的不稳定性和所得策略对噪声的敏感性而著称。已有多种方法被提出以解决这些问题,总而言之,RL 中主要学习到的策略和价值函数的平滑性有助于缓解这些问题。然而,若这些函数过度平滑,将丧失表达能力,导致无法获得全局最优解。因此,本文考虑在局部 Lipschitz 连续性约束下的 RL,即 L2C2。通过基于每个时间步的状态转移为 L2C2 设计时空局部紧致空间,可在不损失表达能力的情况下实现适度平滑。含噪声数值仿真验证了所提 L2C2 在平滑由所学策略生成的机器人动作的同时,提升了任务性能。

英文摘要:

This paper proposes a new regularization technique for reinforcement learning (RL) towards making policy and value functions smooth and stable. RL is known for the instability of the learning process and the sensitivity of the acquired policy to noise. Several methods have been proposed to resolve these problems, and in summary, the smoothness of policy and value functions learned mainly in RL contributes to these problems. However, if these functions are extremely smooth, their expressiveness would be lost, resulting in not obtaining the global optimal solution. This paper therefore considers RL under local Lipschitz continuity constraint, so-called L2C2. By designing the spatio-temporal locally compact space for L2C2 from the state transition at each time step, the moderate smoothness can be achieved without loss of expressiveness. Numerical noisy simulations verified that the proposed L2C2 outperforms the task performance while smoothing out the robot action generated from the learned policy.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑