arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.09807cs.AI
收藏

基于世界模型的无重置强化学习

Reset-free Reinforcement Learning with World Models

The Leiden Institute of Advanced Computer Science(莱顿高级计算机科学研究所);Leiden University(莱顿大学);University of Pennsylvania(宾夕法尼亚大学)

Zhao Yang, Thomas M. Moerland, Mike Preuss, Aske Plaat, Edward S. Hu

AI总结:

针对强化学习训练需大量人力重置的问题,提出基于模型的无重置智能体MoReFree,通过适配探索与策略学习机制优先关注任务相关状态,在无奖励无演示的无重置任务中数据效率更优,显著超越需监督的基线方法,为降低RL人力成本提供新方向。

摘要:

强化学习(RL)是一种极具吸引力的智能体训练范式,能够通过智能体自主获取的经验学习策略。然而,RL的训练过程远未实现自动化,需要大量人力来重置智能体与环境。为解决颇具挑战性的无重置场景问题,我们首先证明了基于模型(MB)的RL方法在该场景下的优势,表明对MBRL的直接适配就能超越所有现有最优方法,且所需监督更少。随后我们指出了这种直接扩展方法的固有局限性,并提出了名为model-based reset-free(MoReFree,基于模型的无重置)智能体的解决方案,以进一步提升性能。MoReFree适配了探索与策略学习两大关键机制,通过优先关注与任务相关的状态来处理无重置任务。它在无需环境奖励或演示的情况下,在各类无重置任务中展现出优异的数据效率,且性能显著优于需要监督的特权基线。我们的研究表明,基于模型的方法在减少RL中的人力投入方面具有巨大潜力。项目主页:https://yangzhao-666.github.io/morefree

英文摘要:

Reinforcement learning (RL) is an appealing paradigm for training intelligent agents, enabling policy acquisition from the agent's own autonomously acquired experience. However, the training process of RL is far from automatic, requiring extensive human effort to reset the agent and environments. To tackle the challenging reset-free setting, we first demonstrate the superiority of model-based (MB) RL methods in such setting, showing that a straightforward adaptation of MBRL can outperform all the prior state-of-the-art methods while requiring less supervision. We then identify limitations inherent to this direct extension and propose a solution called model-based reset-free (MoReFree) agent, which further enhances the performance. MoReFree adapts two key mechanisms, exploration and policy learning, to handle reset-free tasks by prioritizing task-relevant states. It exhibits superior data-efficiency across various reset-free tasks without access to environmental reward or demonstrations while significantly outperforming privileged baselines that require supervision. Our findings suggest model-based methods hold significant promise for reducing human effort in RL. Website: https://yangzhao-666.github.io/morefree

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑