arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2302.09676cs.LG
收藏

利用价值函数的双侧界在强化学习中利用先验知识

Leveraging Prior Knowledge in Reinforcement Learning via Double-Sided Bounds on the Value Function

University of Massachusetts Boston(马萨诸塞大学波士顿分校)

Jacob Adamczyk, Stas Tiomkin, Rahul Kulkarni

AI总结:

本文提出利用价值函数的任意近似推导最优价值函数的双侧界,并扩展至连续空间,从而为训练中的裁剪提供新方法,在简单领域得到验证。

摘要:

智能体利用过去经验的能力对于高效解决新任务至关重要。正如迁移学习、课程学习和组合性研究所证明的那样,新任务的近似解可以从先前推导的价值函数中获得。然而,先前的工作主要集中在使用价值函数来获得新任务解决方案的零样本近似。在这项工作中,我们展示了如何利用价值函数的任意近似来推导出感兴趣的最优价值函数的双侧界。我们进一步通过连续状态和动作空间的误差分析扩展了该框架。推导出的结果导致了训练期间裁剪的新方法,我们在简单领域中进行了数值验证。

英文摘要:

An agent's ability to leverage past experience is critical for efficiently solving new tasks. Approximate solutions for new tasks can be obtained from previously derived value functions, as demonstrated by research on transfer learning, curriculum learning, and compositionality. However, prior work has primarily focused on using value functions to obtain zero-shot approximations for solutions to a new task. In this work, we show how an arbitrary approximation for the value function can be used to derive double-sided bounds on the optimal value function of interest. We further extend the framework with error analysis for continuous state and action spaces. The derived results lead to new approaches for clipping during training which we validate numerically in simple domains.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑