基于对抗网络的模型驱动离线策略优化
Model-based Offline Policy Optimization with Adversarial Network
AI总结：
提出基于对抗网络的模型驱动离线策略优化框架MOAN，利用对抗学习构建泛化能力更强的转移模型并量化不确定性，在离线RL基准上超越现有最先进方法。
摘要：
基于模型的离线强化学习（RL）通过利用日志数据集构建一个监督转移模型，以避免与在线环境进行昂贵的交互，已成为离线策略优化的一种有前景的方法。由于日志数据与在线环境之间的差异可能导致分布偏移问题，许多先前的工作研究了如何保守地构建鲁棒的转移模型并准确估计模型的不确定性。然而，过度保守会限制智能体的探索，且不确定性估计可能不可靠。在这项工作中，我们提出了一种新颖的基于对抗网络的模型驱动离线策略优化框架（MOAN）。其关键思想是利用对抗学习构建一个具有更好泛化能力的转移模型，其中引入一个对抗器来区分分布内样本和分布外样本。此外，对抗器可以自然地提供具有理论保证的模型不确定性量化。大量实验表明，我们的方法在广泛研究的离线RL基准上优于现有的最先进基线。它还能生成多样化的分布内样本，并更准确地量化不确定性。
英文摘要：
Model-based offline reinforcement learning (RL), which builds a supervised transition model with logging dataset to avoid costly interactions with the online environment, has been a promising approach for offline policy optimization. As the discrepancy between the logging data and online environment may result in a distributional shift problem, many prior works have studied how to build robust transition models conservatively and estimate the model uncertainty accurately. However, the over-conservatism can limit the exploration of the agent, and the uncertainty estimates may be unreliable. In this work, we propose a novel Model-based Offline policy optimization framework with Adversarial Network (MOAN). The key idea is to use adversarial learning to build a transition model with better generalization, where an adversary is introduced to distinguish between in-distribution and out-of-distribution samples. Moreover, the adversary can naturally provide a quantification of the model's uncertainty with theoretical guarantees. Extensive experiments showed that our approach outperforms existing state-of-the-art baselines on widely studied offline RL benchmarks. It can also generate diverse in-distribution samples, and quantify the uncertainty more accurately.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
