arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.04253cs.LGcs.AIcs.RO
收藏

Growing Q-Networks:用自适应控制分辨率解决连续控制任务

Growing Q-Networks: Solving Continuous Control Tasks with Adaptive Control Resolution

MIT CSAIL(麻省理工学院计算机科学与人工智能实验室);ISEE AI(ISEE人工智能公司);Google DeepMind(谷歌DeepMind)

Tim Seyde, Peter Werner, Wilko Schwarting, Markus Wulfmeier, Daniela Rus

AI总结:

本文提出 Growing Q-Networks,通过从粗到细自适应增长离散动作空间并结合解耦 Q 学习与价值分解,在最高 dim(A)=38 的连续控制任务中兼顾探索、平滑控制与高性能。

摘要:

近期强化学习方法已显示出 bang-bang(开关式)策略在解决连续控制基准任务方面惊人的强大能力。其底层的粗粒度动作空间离散化通常会带来有利的探索特性,同时根据最优控制理论,在没有动作惩罚的情况下,最终性能不会受到明显损害。在机器人应用中,为减少系统磨损并提高能效,通常更偏好平滑的控制信号,但动作成本可能会损害早期训练阶段的探索。在本工作中,我们旨在通过将离散动作空间从粗控制分辨率逐步增长到细控制分辨率来弥合这一性能差距,并利用 decoupled Q-learning(解耦 Q 学习)的最新成果,将我们的方法扩展到维度高达 dim(A)=38 的高维动作空间。我们的工作表明,自适应控制分辨率与价值分解相结合,可产生简单的仅评论员算法,并在连续控制任务上取得惊人强大的性能。

英文摘要:

Recent reinforcement learning approaches have shown surprisingly strong capabilities of bang-bang policies for solving continuous control benchmarks. The underlying coarse action space discretizations often yield favourable exploration characteristics while final performance does not visibly suffer in the absence of action penalization in line with optimal control theory. In robotics applications, smooth control signals are commonly preferred to reduce system wear and energy efficiency, but action costs can be detrimental to exploration during early training. In this work, we aim to bridge this performance gap by growing discrete action spaces from coarse to fine control resolution, taking advantage of recent results in decoupled Q-learning to scale our approach to high-dimensional action spaces up to dim(A) = 38. Our work indicates that an adaptive control resolution in combination with value decomposition yields simple critic-only algorithms that yield surprisingly strong performance on continuous control tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑