arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1604.02080cs.AIcs.SYeess.SY
收藏

在马尔可夫决策过程中的信息处理约束与模型不确定性规划

Planning with Information-Processing Constraints and Model Uncertainty in Markov Decision Processes

Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所);Max Planck Institute for Biological Cybernetics(马克斯·普朗克生物 cybernetics 研究所);Graduate Training Centre for Neuroscience(神经科学研究生培训中心)

Jordi Grau-Moya, Felix Leibfried, Tim Genewein, Daniel A. Braun

AI总结:

本文提出考虑模型不确定性的马尔可夫决策过程规划方法,通过信息论原理统一解决信息处理约束问题,结合广义变分原理推导价值迭代方案,并在网格世界模拟中验证其有效性。

摘要:

信息论原理已被用于解决特定类别的马尔可夫决策问题。数学上,此类方法由变分自由能原理支配,允许通过相对于参考分布的KL散度表达的信息处理约束来解决MDP规划问题。本文考虑了此类MDP规划器的推广,即考虑模型不确定性。由于模型不确定性也可以形式化为信息处理约束,因此可以从单一广义变分原理中推导出统一的解决方案。我们提供了广义价值迭代方案并给出了收敛性证明。作为极限情况,该广义方案包括标准价值迭代(已知模型)、贝叶斯MDP规划和鲁棒规划。我们在网格世界模拟中展示了该方法的优势。

英文摘要:

Information-theoretic principles for learning and acting have been proposed to solve particular classes of Markov Decision Problems. Mathematically, such approaches are governed by a variational free energy principle and allow solving MDP planning problems with information-processing constraints expressed in terms of a Kullback-Leibler divergence with respect to a reference distribution. Here we consider a generalization of such MDP planners by taking model uncertainty into account. As model uncertainty can also be formalized as an information-processing constraint, we can derive a unified solution from a single generalized variational principle. We provide a generalized value iteration scheme together with a convergence proof. As limit cases, this generalized scheme includes standard value iteration with a known model, Bayesian MDP planning, and robust planning. We demonstrate the benefits of this approach in a grid world simulation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑