基于策略驱动的世界模型适应的鲁棒性离线模型驱动强化学习
Policy-Driven World Model Adaptation for Robust Offline Model-based Reinforcement Learning
AI总结：
本文提出了一种基于策略驱动的世界模型适应框架，通过最大化最小优化问题和Stackelberg学习动态提升离线模型驱动强化学习的鲁棒性。
摘要：
离线强化学习（RL）提供了一种强大的数据驱动控制范式。与无模型方法相比，离线模型驱动RL（MBRL）明确地从静态数据集中学习世界模型，并将其用作替代模拟器，从而提高数据效率，并有可能在数据集支持之外实现泛化。然而，大多数现有的离线MBRL方法遵循两阶段训练过程：首先通过最大化观察转换的似然性学习世界模型，然后优化策略以在学习的模型下最大化其预期回报。这种目标不匹配导致学习的世界模型不一定优化于有效的策略学习。此外，我们发现通过离线MBRL学习的策略在部署过程中往往缺乏鲁棒性，环境中的小对抗性噪声可能导致显著的性能下降。为了解决这些问题，我们提出了一种框架，该框架在统一的学习目标下动态适应世界模型和策略，以提高鲁棒性。我们方法的核心是一个最大化最小优化问题，我们通过创新地利用Stackelberg学习动态来解决。我们提供了理论分析来支持我们的设计，并介绍了计算高效的实现。我们在十二个嘈杂的D4RL MuJoCo任务和三个随机Tokamak控制任务上基准测试了我们的算法，展示了其最先进的性能。
英文摘要：
Offline reinforcement learning (RL) offers a powerful paradigm for data-driven control. Compared to model-free approaches, offline model-based RL (MBRL) explicitly learns a world model from a static dataset and uses it as a surrogate simulator, improving data efficiency and enabling potential generalization beyond the dataset support. However, most existing offline MBRL methods follow a two-stage training procedure: first learning a world model by maximizing the likelihood of the observed transitions, then optimizing a policy to maximize its expected return under the learned model. This objective mismatch results in a world model that is not necessarily optimized for effective policy learning. Moreover, we observe that policies learned via offline MBRL often lack robustness during deployment, and small adversarial noise in the environment can lead to significant performance degradation. To address these, we propose a framework that dynamically adapts the world model alongside the policy under a unified learning objective aimed at improving robustness. At the core of our method is a maximin optimization problem, which we solve by innovatively utilizing Stackelberg learning dynamics. We provide theoretical analysis to support our design and introduce computationally efficient implementations. We benchmark our algorithm on twelve noisy D4RL MuJoCo tasks and three stochastic Tokamak Control tasks, demonstrating its state-of-the-art performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
