In this paper, we present a novel maximum entropy formulation of the Differential Dynamic Programming algorithm and derive two variants using unimodal and multimodal value functions parameterizations. By combining the maximum entropy Bellman equations with a particular approximation of the cost function, we are able to obtain a new formulation of Differential Dynamic Programming which is able to escape from local minima via exploration with a multimodal policy. To demonstrate the efficacy of the proposed algorithm, we provide experimental results using four systems on tasks that are represented by cost functions with multiple local minima and compare them against vanilla Differential Dynamic Programming. Furthermore, we discuss connections with previous work on the linearly solvable stochastic control framework and its extensions in relation to compositionality.
On Certain Hypotheses in Optimal Control Theory and the Relationship of the Maximum Principle with the Dynamic Programming Method Proposed by L. I. Rozonoer
本文将研究L. I. Rozonoer（Automation and Remote Control，2003，vol.64，no.8，pp.1237- 1240）中的最优控制理论，以推导出在所有初始条件下最优控制存在的条件，以及Pontryagin最大值原理与动态规划方法之间的关系。
