离线强化学习中的可证明零样本泛化
Provable Zero-Shot Generalization in Offline Reinforcement Learning
AI总结：
本文针对离线强化学习零样本泛化问题，提出悲观经验风险最小化和悲观近端策略优化两种方法，并证明二者均能找到近最优且可泛化到新环境的策略。
摘要：
在这项工作中，我们研究具有零样本泛化性质（ZSG）的离线强化学习（RL），其中智能体可以访问一个包含来自不同环境经验的离线数据集，智能体的目标是在训练环境上训练一个策略，使其在测试环境中无需进一步交互即可表现良好。现有工作表明，经典离线强化学习无法泛化到新的、未见过的环境。我们提出了悲观经验风险最小化（PERM）和悲观近端策略优化（PPPO），它们利用悲观策略评估来指导策略学习并增强泛化能力。我们证明，PERM 和 PPPO 都能够找到具有 ZSG 的近最优策略。我们的结果是理解离线强化学习中泛化现象基础的第一步。
英文摘要：
In this work, we study offline reinforcement learning (RL) with zero-shot generalization property (ZSG), where the agent has access to an offline dataset including experiences from different environments, and the goal of the agent is to train a policy over the training environments which performs well on test environments without further interaction. Existing work showed that classical offline RL fails to generalize to new, unseen environments. We propose pessimistic empirical risk minimization (PERM) and pessimistic proximal policy optimization (PPPO), which leverage pessimistic policy evaluation to guide policy learning and enhance generalization. We show that both PERM and PPPO are capable of finding a near-optimal policy with ZSG. Our result serves as a first step in understanding the foundation of the generalization phenomenon in offline reinforcement learning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
