arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2301.08502cs.LG
收藏

计划预测:学习基于模型的强化学习的不确定性预测模型

Plan To Predict: Learning an Uncertainty-Foreseeing Model for Model-Based Reinforcement Learning

Sun Yat-sen University(中山大学);Huawei(华为)

Zifan Wu, Chao Yu, Chen Chen, Jianye Hao, Hankz Hankui Zhuo

摘要:

在基于模型的强化学习(MBRL)中,模型学习是至关重要的,因为不准确的模型可以通过生成误导性样本来偏差策略学习。然而,学习准确的模型可能是困难的,因为策略不断更新,并且用于模型学习的访问状态的诱导分布相应地发生变化。现有方法通过量化模型生成的样本的不确定性来缓解这个问题。然而,这些方法只是被动地量化的不确定性后,产生的样本,而不是在模型轨迹落入这些高度不确定的区域之前,消除不确定性。由此产生的低质量样本可能会导致学习目标不稳定,并阻碍策略的优化。此外,虽然被学习以最小化一步预测误差,但模型通常用于预测多个步骤,导致模型学习和模型使用的目标之间的不匹配。为此,我们提出了\emph{Plan To Predict}(P2P),一个MBRL框架,将模型推出过程作为一个连续的决策问题,考虑模型作为决策者和当前的政策作为动态。通过这种方式,模型可以快速适应当前策略,并在生成轨迹时预见未来多步的不确定性。理论上,我们证明了P2P的性能可以通过近似优化真实环境回报的下限来保证。实证结果表明,P2P在几个具有挑战性的基准任务上达到了最先进的性能。

英文摘要:

In Model-based Reinforcement Learning (MBRL), model learning is critical since an inaccurate model can bias policy learning via generating misleading samples. However, learning an accurate model can be difficult since the policy is continually updated and the induced distribution over visited states used for model learning shifts accordingly. Prior methods alleviate this issue by quantifying the uncertainty of model-generated samples. However, these methods only quantify the uncertainty passively after the samples were generated, rather than foreseeing the uncertainty before model trajectories fall into those highly uncertain regions. The resulting low-quality samples can induce unstable learning targets and hinder the optimization of the policy. Moreover, while being learned to minimize one-step prediction errors, the model is generally used to predict for multiple steps, leading to a mismatch between the objectives of model learning and model usage. To this end, we propose \emph{Plan To Predict} (P2P), an MBRL framework that treats the model rollout process as a sequential decision making problem by reversely considering the model as a decision maker and the current policy as the dynamics. In this way, the model can quickly adapt to the current policy and foresee the multi-step future uncertainty when generating trajectories. Theoretically, we show that the performance of P2P can be guaranteed by approximately optimizing a lower bound of the true environment return. Empirical results demonstrate that P2P achieves state-of-the-art performance on several challenging benchmark tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑