arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2208.14501cs.LGcs.AIcs.ROcs.SYeess.SY
收藏

基于SINDy的基于模型的强化学习

Model-Based Reinforcement Learning with SINDy

Manning College of Information and Computer Sciences(曼宁信息与计算机科学学院);University of Massachusetts Amherst(马萨诸塞大学阿默斯特分校)

Rushiv Arora, Bruno Castro da Silva, Eliot Moss

摘要:

我们利用物理学界的最新进展,提出了一种新的方法,用于发现强化学习(RL)中物理系统的非线性动力学。我们建立,这种方法是能够发现使用显着更少的轨迹(少到一个推出$\leq 30$时间步长)比最先进的模型学习算法的基础动态。此外,该技术学习一个模型,该模型足够准确,可以在比无模型算法所需的轨迹少得多的情况下诱导出接近最优的策略。它带来了基于模型的强化学习的好处,而不需要提前开发模型,用于具有基于物理的动态系统。为了验证该算法的有效性和适用性,我们对四个经典控制任务进行了实验。我们发现,一个最优的政策训练的基础系统的发现动态可以推广。此外,学习的策略在实际物理系统上部署时表现良好,从而弥合了模型与实际系统之间的差距。我们进一步比较我们的方法,最先进的基于模型和无模型的方法,并表明我们的方法需要更少的轨迹采样的真实物理系统相比,其他方法。此外,我们探索了近似动力学模型,发现它们也可以表现得很好。

英文摘要:

We draw on the latest advancements in the physics community to propose a novel method for discovering the governing non-linear dynamics of physical systems in reinforcement learning (RL). We establish that this method is capable of discovering the underlying dynamics using significantly fewer trajectories (as little as one rollout with $\leq 30$ time steps) than state of the art model learning algorithms. Further, the technique learns a model that is accurate enough to induce near-optimal policies given significantly fewer trajectories than those required by model-free algorithms. It brings the benefits of model-based RL without requiring a model to be developed in advance, for systems that have physics-based dynamics. To establish the validity and applicability of this algorithm, we conduct experiments on four classic control tasks. We found that an optimal policy trained on the discovered dynamics of the underlying system can generalize well. Further, the learned policy performs well when deployed on the actual physical system, thus bridging the model to real system gap. We further compare our method to state-of-the-art model-based and model-free approaches, and show that our method requires fewer trajectories sampled on the true physical system compared other methods. Additionally, we explored approximate dynamics models and found that they also can perform well.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑