arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1902.07664math.OC
收藏

利用广义Benders割学习连续Q函数

Learning continuous Q-Functions using generalized Benders cuts

Joseph Warrington

摘要:

Q函数广泛应用于离散时间学习和控制中,当给定初态和输入时,对给定控制策略所产生的未来成本进行建模。虽然Q函数的一些性质已经被理解,但为连续问题生成最优策略的Q函数通常很难计算。即使当系统模型可用时,最优控制通常也难以实现,除非在极少数情况下恰好存在解析解,或者可以计算出显式精确解。通常需要离散状态和动作空间,或者用难以先验选择的基来参数化Q函数。本文描述了一种基于模型的算法,基于广义Benders理论,产生越来越紧密的外近似的最佳Q函数。在强对偶假设下,我们证明了该算法在状态输入空间中的任意有限个点处,在有限次迭代中产生任意小的Bellman最优性误差。在额外的假设下,当输入由算法的更新Q函数在线确定时,同样的保证成立。我们证明了这些性质的数值标量和8维系统。

英文摘要:

Q-functions are widely used in discrete-time learning and control to model future costs arising from a given control policy, when the initial state and input are given. Although some of their properties are understood, Q-functions generating optimal policies for continuous problems are usually hard to compute. Even when a system model is available, optimal control is generally difficult to achieve except in rare cases where an analytical solution happens to exist, or an explicit exact solution can be computed. It is typically necessary to discretize the state and action spaces, or parameterize the Q-function with a basis that can be hard to select a priori. This paper describes a model-based algorithm based on generalized Benders theory that yields ever-tighter outer-approximations of the optimal Q-function. Under a strong duality assumption, we prove that the algorithm yields an arbitrarily small Bellman optimality error at any finite number of arbitrary points in the state-input space, in finite iterations. Under additional assumptions, the same guarantee holds when the inputs are determined online by the algorithm's updating Q-function. We demonstrate these properties numerically on scalar and 8-dimensional systems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑