arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.00290cs.RO
收藏

面向机器人应用的可变时间步长强化学习

Variable Time Step Reinforcement Learning for Robotic Applications

Polytechnique Montréal(蒙特利尔理工学院)

Dong Wang, Giovanni Beltrame

AI总结:

本文提出MOSEAC方法实现可变时间步长强化学习,通过自适应控制频率在必要时才执行动作,并在仿真和真实机器人上验证了其收敛更快、训练效果更好且能耗更低。

摘要:

传统强化学习(RL)生成离散控制策略,每个周期分配一个动作。这些策略通常以固定频率控制回路的形式实现。这种刚性带来了挑战,因为最优控制频率依赖于具体任务;次优频率会增加计算需求并降低探索效率。可变时间步长强化学习(VTS-RL)通过自适应控制频率来解决这些问题,仅在必要时执行动作,从而降低计算负载,并将动作空间扩展为包含动作持续时间。在本文中,我们引入了多目标软弹性演员-评论家(MOSEAC)方法来实现VTS-RL,并通过理论分析以及在仿真和真实机器人上的实验对其进行验证。结果表明,与其他可变频率或固定频率方法相比,该方法收敛更快、训练效果更好,并且能耗更低。

英文摘要:

Traditional reinforcement learning (RL) generates discrete control policies, assigning one action per cycle. These policies are usually implemented as in a fixed-frequency control loop. This rigidity presents challenges as optimal control frequency is task-dependent; suboptimal frequencies increase computational demands and reduce exploration efficiency. Variable Time Step Reinforcement Learning (VTS-RL) addresses these issues with adaptive control frequencies, executing actions only when necessary, thus reducing computational load and extending the action space to include action durations. In this paper we introduce the Multi-Objective Soft Elastic Actor-Critic (MOSEAC) method to perform VTS-RL, validating it through theoretical analysis and experimentation in simulation and on real robots. Results show faster convergence, better training results, and reduced energy consumption with respect to other variable- or fixed-frequency approaches.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑