arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2105.07490math.OC
收藏

部分可观测马氏决策过程的熵最大化

Entropy Maximization for Partially Observable Markov Decision Processes

Yagiz Savas, Michael Hibbard, Bo Wu, Takashi Tanaka, Ufuk Topcu

摘要:

我们研究的问题,合成的控制器,最大化的熵的部分可观察的马尔可夫决策过程(POMDP)的约束下的预期总回报。这样的控制器最大限度地减少了一个代理的轨迹的可预测性,以外部观察员,同时保证完成一个任务所表示的报酬函数。我们首先证明了一个部分观察的代理可以实现熵最多以及一个代理完美的意见。然后,专注于有限状态控制器(FSC)与确定性的记忆转换,我们表明,最大熵的POMDP是下界的参数马尔可夫链(pMC)诱导的FSC的最大熵。这种关系使我们能够重铸的熵最大化问题作为一个所谓的参数合成问题的诱导pMC。然后,我们提出了一种算法来合成一个FSC,本地最大化的POMDP的FSC与相同数量的内存状态的熵。在数值例子中,我们说明了最大熵,在FSC的记忆状态的数量,和预期的回报之间的关系。

英文摘要:

We study the problem of synthesizing a controller that maximizes the entropy of a partially observable Markov decision process (POMDP) subject to a constraint on the expected total reward. Such a controller minimizes the predictability of an agent's trajectories to an outside observer while guaranteeing the completion of a task expressed by a reward function. We first prove that an agent with partial observations can achieve an entropy at most as well as an agent with perfect observations. Then, focusing on finite-state controllers (FSCs) with deterministic memory transitions, we show that the maximum entropy of a POMDP is lower bounded by the maximum entropy of the parametric Markov chain (pMC) induced by such FSCs. This relationship allows us to recast the entropy maximization problem as a so-called parameter synthesis problem for the induced pMC. We then present an algorithm to synthesize an FSC that locally maximizes the entropy of a POMDP over FSCs with the same number of memory states. In numerical examples, we illustrate the relationship between the maximum entropy, the number of memory states in the FSC, and the expected reward.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑