arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2301.02328cs.LGcs.AIcs.RO
收藏

极限Q学习:无熵的MaxEnt RL

Extreme Q-Learning: MaxEnt RL without Entropy

Stanford University(斯坦福大学);Google Brain(谷歌大脑)

Divyansh Garg, Joey Hejna, Matthieu Geist, Stefano Ermon

摘要:

现代深度强化学习(RL)算法需要估计最大Q值,这在具有无限数量可能动作的连续域中很难计算。在这项工作中,我们引入了一个新的在线和离线RL的更新规则,直接使用极值理论(EVT)的最大值模型,从经济学中汲取灵感。通过这样做,我们避免了使用分布外操作来计算Q值,这通常是错误的重要来源。我们的关键见解是引入一个目标,直接估计最大熵RL设置中的最佳软价值函数(LogSumExp),而不需要从策略中采样。使用EVT,我们得到我们的\emph{Extreme Q-Learning}框架,因此在线,并首次离线MaxEnt Q-学习算法,不明确要求访问的政策或其熵。我们的方法在D4 RL基准测试中获得了一致的强劲性能,在具有挑战性的Franka Kitchen任务上优于\emph{10+ points}的先前工作,同时在在线DM控制任务上提供了SAC和TD 3的适度改进。可视化和代码可以在我们的网站https://div99.github.io/XQL/上找到。

英文摘要:

Modern Deep Reinforcement Learning (RL) algorithms require estimates of the maximal Q-value, which are difficult to compute in continuous domains with an infinite number of possible actions. In this work, we introduce a new update rule for online and offline RL which directly models the maximal value using Extreme Value Theory (EVT), drawing inspiration from economics. By doing so, we avoid computing Q-values using out-of-distribution actions which is often a substantial source of error. Our key insight is to introduce an objective that directly estimates the optimal soft-value functions (LogSumExp) in the maximum entropy RL setting without needing to sample from a policy. Using EVT, we derive our \emph{Extreme Q-Learning} framework and consequently online and, for the first time, offline MaxEnt Q-learning algorithms, that do not explicitly require access to a policy or its entropy. Our method obtains consistently strong performance in the D4RL benchmark, outperforming prior works by \emph{10+ points} on the challenging Franka Kitchen tasks while offering moderate improvements over SAC and TD3 on online DM Control tasks. Visualizations and code can be found on our website at https://div99.github.io/XQL/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑