面向离线强化学习的无监督数据生成：基于模型的视角
Unsupervised Data Generation for Offline Reinforcement Learning: A Perspective from Model
AI总结：
本文从基于模型的离线强化学习优化视角，理论分析了批次数据对算法性能的影响，提出UDG方法在任务无关设置下生成并选择数据，有效缩小与最优策略的性能差距并在未知任务上超越监督式数据生成。
摘要：
离线强化学习（RL）近期引起了RL研究者日益增长的兴趣。然而，离线RL的性能受到分布外问题的困扰，而这一问题在在线RL中可以通过反馈来纠正。以往的离线RL研究侧重于将离线算法限制在分布内甚至样本内的动作采样。相比之下，较少有工作关注批次数据的影响。在本文中，我们首先从基于模型的离线RL优化的角度，在理论上为批次数据和离线RL算法的性能之间搭建了一座桥梁。我们得出结论，在温和的假设下，由行为策略生成的状态-动作对分布与最优策略生成的分布之间的距离，解释了基于模型的离线RL学习到的策略与最优策略之间的性能差距。其次，我们揭示了在任务无关设置中，由无监督RL训练的一系列策略可以最小化性能差距中的最坏情况遗憾。受理论结论的启发，UDG（无监督数据生成）被设计用于在任务无关设置下生成数据并为离线训练选择合适的数据。实证结果表明，UDG在解决未知任务上能够超越监督式数据生成。
英文摘要：
Offline reinforcement learning (RL) recently gains growing interests from RL researchers. However, the performance of offline RL suffers from the out-of-distribution problem, which can be corrected by feedback in online RL. Previous offline RL research focuses on restricting the offline algorithm in in-distribution even in-sample action sampling. In contrast, fewer work pays attention to the influence of the batch data. In this paper, we first build a bridge over the batch data and the performance of offline RL algorithms theoretically, from the perspective of model-based offline RL optimization. We draw a conclusion that, with mild assumptions, the distance between the state-action pair distribution generated by the behavioural policy and the distribution generated by the optimal policy, accounts for the performance gap between the policy learned by model-based offline RL and the optimal policy. Secondly, we reveal that in task-agnostic settings, a series of policies trained by unsupervised RL can minimize the worst-case regret in the performance gap. Inspired by the theoretical conclusions, UDG (Unsupervised Data Generation) is devised to generate data and select proper data for offline training under tasks-agnostic settings. Empirical results demonstrate that UDG can outperform supervised data generation on solving unknown tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
