arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.00987cs.LG
收藏

S$^2$AC:基于能量的强化学习与 Stein 软演员-评论家

S$^2$AC: Energy-Based Reinforcement Learning with Stein Soft Actor Critic

Qatar Computing Research Institute, Hamad Bin Khalifa University(卡塔尔计算研究所、哈马德·本·哈利法大学);School of Computer Science and Engineering, Nanyang Technological University(南洋理工大学计算机科学与工程学院);SalesForce(SalesForce(赛富时));Skywork AI;Data Science, William & Mary(威廉玛丽学院数据科学系)

Safa Messaoud, Billel Mokeddem, Zhenghai Xue, Linsey Pang, Bo An, Haipeng Chen, Sanjay Chawla

AI总结:

针对最大熵强化学习中能量模型熵估计困难的问题,提出 Stein 软演员-评论家算法,利用参数化 SVGD 作为策略并推导熵的闭式高效表达式,在 MuJoCo 等任务上优于 SAC 和 SQL。

摘要:

学习具有表达能力的随机策略而非确定性策略,已被提出用于获得更好的稳定性、样本复杂度和鲁棒性。值得注意的是,在最大熵强化学习(MaxEnt RL)中,策略被建模为基于 Q 值的具有表达能力的能量模型(EBM)。然而,这种形式化方法需要对这类 EBM 的熵进行估计,而这是一个尚未解决的问题。为了解决这一问题,以往的 MaxEnt RL 方法要么隐式地估计熵,导致较高的计算复杂度和方差(SQL),要么遵循变分推断过程,拟合简化的演员分布(例如高斯分布)以保证可处理性(SAC)。我们提出 Stein 软演员-评论家(S$^2$AC),这是一种在不牺牲效率的情况下学习具有表达能力的策略的 MaxEnt RL 算法。具体而言,S$^2$AC 使用参数化的 Stein 变分梯度下降(SVGD)作为底层策略。我们推导出这类策略熵的闭式表达式。我们的公式计算效率高,并且仅依赖于一阶导数和向量积。实验结果表明,在多目标环境中,S$^2$AC 比 SQL 和 SAC 得到更优的 MaxEnt 目标解,并且在 MuJoCo 基准上优于 SAC 和 SQL。我们的代码可在以下地址获取:https://github.com/SafaMessaoud/S2AC-Energy-Based-RL-with-Stein-Soft-Actor-Critic

英文摘要:

Learning expressive stochastic policies instead of deterministic ones has been proposed to achieve better stability, sample complexity, and robustness. Notably, in Maximum Entropy Reinforcement Learning (MaxEnt RL), the policy is modeled as an expressive Energy-Based Model (EBM) over the Q-values. However, this formulation requires the estimation of the entropy of such EBMs, which is an open problem. To address this, previous MaxEnt RL methods either implicitly estimate the entropy, resulting in high computational complexity and variance (SQL), or follow a variational inference procedure that fits simplified actor distributions (e.g., Gaussian) for tractability (SAC). We propose Stein Soft Actor-Critic (S$^2$AC), a MaxEnt RL algorithm that learns expressive policies without compromising efficiency. Specifically, S$^2$AC uses parameterized Stein Variational Gradient Descent (SVGD) as the underlying policy. We derive a closed-form expression of the entropy of such policies. Our formula is computationally efficient and only depends on first-order derivatives and vector products. Empirical results show that S$^2$AC yields more optimal solutions to the MaxEnt objective than SQL and SAC in the multi-goal environment, and outperforms SAC and SQL on the MuJoCo benchmark. Our code is available at: https://github.com/SafaMessaoud/S2AC-Energy-Based-RL-with-Stein-Soft-Actor-Critic

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑