贝叶斯自适应蒙特卡洛树搜索用于离线模型驱动强化学习
Bayes Adaptive Monte Carlo Tree Search for Offline Model-based Reinforcement Learning
AI总结：
本文提出贝叶斯自适应蒙特卡洛树搜索算法，用于提升离线模型驱动强化学习的性能，显著优于现有方法。
摘要：
离线强化学习（RL）是一种用于数据驱动决策和控制的强大方法。与无模型方法相比，离线模型驱动强化学习（MBRL）明确地从静态数据集中学习世界模型，并将其用作替代模拟器，从而提高数据效率，并使学习到的策略有可能超越数据集的支持。然而，可能存在各种马尔可夫决策过程（MDP），它们在离线数据集上表现相同，处理真实MDP的不确定性可能具有挑战性。在本文中，我们提出将离线MBRL建模为贝叶斯自适应马尔可夫决策过程（BAMDP），这是一个解决模型不确定性的原则性框架。我们进一步提出了一种新颖的贝叶斯自适应蒙特卡洛规划算法，能够解决具有随机转移的连续状态和动作空间中的BAMDP。该规划过程基于蒙特卡洛树搜索，并可以集成到离线MBRL中作为策略改进算子在策略迭代中。我们提出的
英文摘要：
Offline reinforcement learning (RL) is a powerful approach for data-driven decision-making and control. Compared to model-free methods, offline model-based reinforcement learning (MBRL) explicitly learns world models from a static dataset and uses them as surrogate simulators, improving the data efficiency and enabling the learned policy to potentially generalize beyond the dataset support. However, there could be various MDPs that behave identically on the offline dataset and dealing with the uncertainty about the true MDP can be challenging. In this paper, we propose modeling offline MBRL as a Bayes Adaptive Markov Decision Process (BAMDP), which is a principled framework for addressing model uncertainty. We further propose a novel Bayes Adaptive Monte-Carlo planning algorithm capable of solving BAMDPs in continuous state and action spaces with stochastic transitions. This planning process is based on Monte Carlo Tree Search and can be integrated into offline MBRL as a policy improvement operator in policy iteration. Our "RL + Search" framework follows in the footsteps of superhuman AIs like AlphaZero, improving on current offline MBRL methods by incorporating more computation input. The proposed algorithm significantly outperforms state-of-the-art offline RL methods on twelve D4RL MuJoCo tasks and three challenging, stochastic tokamak control tasks. The codebase is available at: https://github.com/LucasCJYSDL/Offline-RL-Kit.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
