arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1708.09096math.OCcs.ITmath.IT
收藏

转移熵正则化马氏决策过程

Transfer-Entropy-Regularized Markov Decision Processes

Takashi Tanaka, Henrik Sandberg, Mikael Skoglund

摘要:

考虑转移熵正则化的马尔可夫决策过程(TERMDP)框架,其中经典状态相关成本与从状态随机过程到控制随机过程的转移熵的加权和最小化。虽然TERMDP一般被制定为非凸的最优化问题,我们推导出一个解析的必要最优性条件表示为一个有限集的非线性方程,基于此迭代的向前向后的计算过程类似的Arimoto-Blahut算法。结果表明,该算法产生的序列的每一个极限点都是TERMDP的平稳点。在网络控制系统理论和非平衡态热力学的背景下讨论了TERMDP的应用。所提出的算法被应用到一个信息约束的迷宫导航问题,从而我们研究如何定性地改变信息的价格的最优决策polices。

英文摘要:

We consider the framework of transfer-entropy-regularized Markov Decision Process (TERMDP) in which the weighted sum of the classical state-dependent cost and the transfer entropy from the state random process to the control random process is minimized. Although TERMDPs are generally formulated as nonconvex optimization problems, we derive an analytical necessary optimality condition expressed as a finite set of nonlinear equations, based on which an iterative forward-backward computational procedure similar to the Arimoto-Blahut algorithm is proposed. It is shown that every limit point of the sequence generated by the proposed algorithm is a stationary point of the TERMDP. Applications of TERMDPs are discussed in the context of networked control systems theory and non-equilibrium thermodynamics. The proposed algorithm is applied to an information-constrained maze navigation problem, whereby we study how the price of information qualitatively alters the optimal decision polices.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑