arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2306.02451cs.LGcs.AIstat.ML
收藏

For SALE:面向深度强化学习的状态-动作表示学习

For SALE: State-Action Representation Learning for Deep Reinforcement Learning

Mila, McGill University(米拉,麦吉尔大学);Google DeepMind(谷歌DeepMind)

Scott Fujimoto, Wei-Di Chang, Edward J. Smith, Shixiang Shane Gu, Doina Precup, David Meger

AI总结:

针对低维状态强化学习中表示学习被忽视的问题,提出建模状态-动作交互的SALE方法,结合检查点机制改进TD3得到TD7算法,在OpenAI gym基准上性能大幅提升且适配在线离线场景。

摘要:

在强化学习(RL)领域,表示学习是处理复杂基于图像的任务的成熟工具,但在低维状态环境(如物理控制问题)中常被忽视。本文提出SALE,一种学习嵌入的新方法,可建模状态与动作之间的细微交互,实现从低维状态中进行有效的表示学习。我们广泛研究了这些嵌入的设计空间,并强调了重要的设计考量。我们将SALE和适配RL的检查点机制集成到TD3中,形成TD7算法,其性能显著优于现有连续控制算法。在OpenAI gym基准任务上,TD7在30万和500万时间步时,性能相比TD3分别平均提升276.7%和50.7%,且在在线和离线设置下均有效。

英文摘要:

In the field of reinforcement learning (RL), representation learning is a proven tool for complex image-based tasks, but is often overlooked for environments with low-level states, such as physical control problems. This paper introduces SALE, a novel approach for learning embeddings that model the nuanced interaction between state and action, enabling effective representation learning from low-level states. We extensively study the design space of these embeddings and highlight important design considerations. We integrate SALE and an adaptation of checkpoints for RL into TD3 to form the TD7 algorithm, which significantly outperforms existing continuous control algorithms. On OpenAI gym benchmark tasks, TD7 has an average performance gain of 276.7% and 50.7% over TD3 at 300k and 5M time steps, respectively, and works in both the online and offline settings.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑