arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2006.05956math.OCcs.LGmath.PR
收藏

正则化随机控制问题的梯度流

Gradient Flows for Regularized Stochastic Control Problems

University of Edinburgh(爱丁堡大学);The Alan Turing Institute(艾伦·图灵研究所)

David Šiška, Łukasz Szpruch

AI总结:

本文为相对熵正则化的测度值随机控制问题构造梯度流,证明其成本下降、不变测度满足 Pontryagin 原理、强凸时指数收敛,并给出 Bayesian 解释以支撑强化学习中随机梯度算法的收敛理论。

摘要:

本文研究以概率测度作为动作空间、目标函数由相对熵进行惩罚的随机控制问题。我们确定了合适的度量空间,并在该空间上于容许控制集合内为测度值控制过程构造了一个梯度流,沿该梯度流成本泛函保证下降。文中证明,该梯度流的任意不变测度都满足 Pontryagin 最优性原理。如果所处理的问题具有足够的凸性,则梯度流以指数速度收敛。此外,最优测度值控制过程具有一种 Bayesian 解释,这意味着在求解此类随机控制问题时可以纳入先验知识。本工作的动机是希望扩展随机梯度类算法收敛性的理论基础;这类算法在强化学习社区中被广泛用于求解控制问题。

英文摘要:

This paper studies stochastic control problems with the action space taken to be probability measures, with the objective penalised by the relative entropy. We identify suitable metric space on which we construct a gradient flow for the measure-valued control process, in the set of admissible controls, along which the cost functional is guaranteed to decrease. It is shown that any invariant measure of this gradient flow satisfies the Pontryagin optimality principle. If the problem we work with is sufficiently convex, the gradient flow converges exponentially fast. Furthermore, the optimal measure-valued control process admits a Bayesian interpretation which means that one can incorporate prior knowledge when solving such stochastic control problems. This work is motivated by a desire to extend the theoretical underpinning for the convergence of stochastic gradient type algorithms widely employed in the reinforcement learning community to solve control problems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑