arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2501.16142cs.LGcs.AI
收藏

迈向通用无模型强化学习

Towards General-Purpose Model-Free Reinforcement Learning

Meta FAIR

Scott Fujimoto, Pierluca D'Oro, Amy Zhang, Yuandong Tian, Michael Rabbat

AI总结:

本文提出统一的无模型深度强化学习算法MR.Q,利用基于模型的表示线性化价值函数以获得密集任务目标,同时避免规划成本,在单一超参数下跨基准表现优异。

摘要:

强化学习(RL)承诺了一个近乎通用的问题解决框架。然而在实践中,RL算法通常针对特定基准量身定制,依赖于精心调整的超参数和算法选择。近期,强大的基于模型的RL方法在跨基准测试中展示了令人印象深刻的通用结果,但代价是复杂性增加和运行时间缓慢,限制了其更广泛的适用性。在本文中,我们试图找到一种统一的无模型深度RL算法,能够处理多样化的领域和问题设置。为此,我们利用近似线性化价值函数的基于模型的表示,借鉴基于模型RL使用的更密集任务目标,同时避免与规划或模拟轨迹相关的成本。我们使用单一超参数集在各种常见RL基准上评估了我们的算法MR.Q,并展示了与特定领域基线和通用基线相媲美的性能,为构建通用无模型深度RL算法提供了具体步骤。

英文摘要:

Reinforcement learning (RL) promises a framework for near-universal problem-solving. In practice however, RL algorithms are often tailored to specific benchmarks, relying on carefully tuned hyperparameters and algorithmic choices. Recently, powerful model-based RL methods have shown impressive general results across benchmarks but come at the cost of increased complexity and slow run times, limiting their broader applicability. In this paper, we attempt to find a unifying model-free deep RL algorithm that can address a diverse class of domains and problem settings. To achieve this, we leverage model-based representations that approximately linearize the value function, taking advantage of the denser task objectives used by model-based RL while avoiding the costs associated with planning or simulated trajectories. We evaluate our algorithm, MR.Q, on a variety of common RL benchmarks with a single set of hyperparameters and show a competitive performance against domain-specific and general baselines, providing a concrete step towards building general-purpose model-free deep RL algorithms.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑