离线强化学习的离线轨迹优化
Offline Trajectory Optimization for Offline Reinforcement Learning
AI总结：
提出OTTO离线轨迹优化方法，通过World Transformers生成长时序模拟轨迹，并利用基于不确定性的World Evaluator评估与校正数据，有效提升了离线RL算法在复杂环境中的性能。
摘要：
离线强化学习（RL）旨在无需在线探索的情况下学习策略。为了扩大训练数据，基于模型的离线RL学习一个动态模型，并将其作为虚拟环境来生成模拟数据和增强策略学习。然而，现有的离线RL数据增强方法存在以下问题：（i）短时序模拟带来的改进微不足道；（ii）缺乏对生成数据的评估和校正，导致低质量的增强。在本文中，我们提出了用于离线强化学习的离线轨迹优化（OTTO）。其核心动机是进行长时序模拟，然后利用模型不确定性来评估和校正增强数据。具体而言，我们提出了一个Transformer集成模型（又称World Transformers），用于预测环境状态动态和奖励函数。我们提出了三种策略，通过扰动离线数据中的动作，使用World Transformers生成长时序轨迹模拟。随后，引入了基于不确定性的World Evaluator，首先评估生成轨迹的置信度，然后对低置信度数据进行校正。最后，我们联合使用原始数据和校正后的增强数据来训练离线RL算法。OTTO作为一个插件模块，可以与现有的无模型离线RL方法集成。在各种基准上的实验表明，OTTO能有效提升代表性离线RL算法的性能，包括在像AntMaze这样具有稀疏奖励的复杂环境中。代码可在https://github.com/ZiqiZhao1/OTTO获取。
英文摘要：
Offline reinforcement learning (RL) aims to learn policies without online explorations. To enlarge the training data, model-based offline RL learns a dynamics model which is utilized as a virtual environment to generate simulation data and enhance policy learning. However, existing data augmentation methods for offline RL suffer from (i) trivial improvement from short-horizon simulation; and (ii) the lack of evaluation and correction for generated data, leading to low-qualified augmentation. In this paper, we propose offline trajectory optimization for offline reinforcement learning (OTTO). The key motivation is to conduct long-horizon simulation and then utilize model uncertainty to evaluate and correct the augmented data. Specifically, we propose an ensemble of Transformers, a.k.a. World Transformers, to predict environment state dynamics and the reward function. Three strategies are proposed to use World Transformers to generate long-horizon trajectory simulation by perturbing the actions in the offline data. Then, an uncertainty-based World Evaluator is introduced to firstly evaluate the confidence of the generated trajectories and then perform the correction for low-confidence data. Finally, we jointly use the original data with the corrected augmentation data to train an offline RL algorithm. OTTO serves as a plug-in module and can be integrated with existing model-free offline RL methods. Experiments on various benchmarks show that OTTO can effectively improve the performance of representative offline RL algorithms, including in complex environments with sparse rewards like AntMaze. Codes are available at https://github.com/ZiqiZhao1/OTTO.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
