离线强化学习的对抗模型
Adversarial Model for Offline Reinforcement Learning
AI总结：
提出ARMOR框架，通过对抗训练MDP模型，在离线强化学习中稳健改进任意参考策略，理论保证性能不降，实践效果与最先进算法相当。
摘要：
我们提出了一种新颖的基于模型的离线强化学习（RL）框架，称为离线强化学习的对抗模型（ARMOR），它能够稳健地学习策略，无论数据覆盖情况如何，都能在任意参考策略的基础上进行改进。ARMOR旨在通过对抗性训练马尔可夫决策过程模型，优化策略以相对于参考策略的最坏情况性能。理论上，我们证明，当参考策略得到数据支持时，ARMOR配合良好调整的超参数，可以与数据覆盖范围内的最佳策略竞争。同时，ARMOR对超参数选择具有鲁棒性：对于“任何”可接受的超参数，ARMOR学习到的策略永远不会降低参考策略的性能，即使参考策略未被数据集覆盖。为了在实践中验证这些性质，我们设计了ARMOR的可扩展实现，通过对抗训练，与典型的基于模型的方法不同，它可以在不使用模型集成的情况下优化策略。我们表明，ARMOR在离线无模型和基于模型的强化学习算法中均达到了与最先进方法相当的性能，并且能够在各种超参数选择下稳健地改进参考策略。
英文摘要：
We propose a novel model-based offline Reinforcement Learning (RL) framework, called Adversarial Model for Offline Reinforcement Learning (ARMOR), which can robustly learn policies to improve upon an arbitrary reference policy regardless of data coverage. ARMOR is designed to optimize policies for the worst-case performance relative to the reference policy through adversarially training a Markov decision process model. In theory, we prove that ARMOR, with a well-tuned hyperparameter, can compete with the best policy within data coverage when the reference policy is supported by the data. At the same time, ARMOR is robust to hyperparameter choices: the policy learned by ARMOR, with "any" admissible hyperparameter, would never degrade the performance of the reference policy, even when the reference policy is not covered by the dataset. To validate these properties in practice, we design a scalable implementation of ARMOR, which by adversarial training, can optimize policies without using model ensembles in contrast to typical model-based methods. We show that ARMOR achieves competent performance with both state-of-the-art offline model-free and model-based RL algorithms and can robustly improve the reference policy over various hyperparameter choices.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
