arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2212.04407cs.LGcs.AI
收藏

具有连续选项的动态决策频率

Dynamic Decision Frequency with Continuous Options

Alberta Machine Intelligence Institute (Amii)(阿尔伯塔机器智能研究所);Department of Computer Science, University of Alberta(阿尔伯塔大学计算机科学系);Department of Computer Science and Digital Science Center, University of Innsbruck(因斯布鲁克大学计算机科学系和数字科学中心)

Amirmohammad Karimi, Jun Jin, Jun Luo, A. Rupam Mahmood, Martin Jagersand, Samuele Tosatto

AI总结:

提出CTCO框架,通过时间连续的变持续时间子策略选项,使智能体能动态调整决策频率,在连续控制任务中不受环境交互频率影响,并有效促进稀疏奖励下的机械臂探索。

摘要:

在经典的强化学习算法中,智能体在离散且固定的时间间隔做出决策。决策之间的持续时间成为一个关键的超参数,因为设置过短可能会增加问题的难度,要求智能体做出大量决策才能实现其目标,而设置过长可能导致智能体失去对系统的控制。然而,物理系统不一定需要恒定的控制频率,对于学习型智能体,通常更倾向于在可能时以低频率运行,在必要时以高频率运行。我们提出了一个名为 Continuous-Time Continuous-Options (CTCO) 的框架,智能体在其中选择作为可变持续时间子策略的选项。这些选项是时间连续的,能够以任何所需频率与系统交互,提供动作的平滑变化。我们通过在具有各种动作周期时间的模拟连续控制任务上,将其性能与经典 RL 和时间抽象 RL 方法进行比较,展示了 CTCO 的有效性。我们表明,我们算法的性能不受环境交互频率选择的影响。此外,我们展示了 CTCO 在促进具有稀疏奖励的 7 自由度机械臂真实世界视觉到达任务探索方面的有效性。

英文摘要:

In classic reinforcement learning algorithms, agents make decisions at discrete and fixed time intervals. The duration between decisions becomes a crucial hyperparameter, as setting it too short may increase the problem's difficulty by requiring the agent to make numerous decisions to achieve its goal while setting it too long can result in the agent losing control over the system. However, physical systems do not necessarily require a constant control frequency, and for learning agents, it is often preferable to operate with a low frequency when possible and a high frequency when necessary. We propose a framework called Continuous-Time Continuous-Options (CTCO), where the agent chooses options as sub-policies of variable durations. These options are time-continuous and can interact with the system at any desired frequency providing a smooth change of actions. We demonstrate the effectiveness of CTCO by comparing its performance to classical RL and temporal-abstraction RL methods on simulated continuous control tasks with various action-cycle times. We show that our algorithm's performance is not affected by the choice of environment interaction frequency. Furthermore, we demonstrate the efficacy of CTCO in facilitating exploration in a real-world visual reaching task for a 7 DOF robotic arm with sparse rewards.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑