连续状态Markov决策过程的核Taylor价值函数逼近
Kernel Taylor-Based Value Function Approximation for Continuous-State Markov Decision Processes
摘要:
我们提出了一个原则性的基于核的策略迭代算法来解决连续状态马尔可夫决策过程(MDP)。与大多数假设完全已知状态转换模型的决策理论规划框架相反,我们设计了一种方法,消除了这样一个强有力的假设,这在现实中通常是非常困难的。为了实现这一点,我们首先应用价值函数的二阶泰勒展开。Bellman最优性方程近似为一个偏微分方程,它只依赖于过渡模型的一阶和二阶矩。结合价值函数的核表示,设计了一种有效的策略迭代算法,其策略评估步骤可以表示为一个由有限个支持状态组成的线性方程组。我们已经验证了所提出的方法,通过广泛的模拟在简化和现实的规划方案,实验表明,我们提出的方法导致了更优越的性能比几个基线方法。
英文摘要:
We propose a principled kernel-based policy iteration algorithm to solve the continuous-state Markov Decision Processes (MDPs). In contrast to most decision-theoretic planning frameworks, which assume fully known state transition models, we design a method that eliminates such a strong assumption, which is oftentimes extremely difficult to engineer in reality. To achieve this, we first apply the second-order Taylor expansion of the value function. The Bellman optimality equation is then approximated by a partial differential equation, which only relies on the first and second moments of the transition model. By combining the kernel representation of value function, we then design an efficient policy iteration algorithm whose policy evaluation step can be represented as a linear system of equations characterized by a finite set of supporting states. We have validated the proposed method through extensive simulations in both simplified and realistic planning scenarios, and the experiments show that our proposed approach leads to a much superior performance over several baseline methods.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。