arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1907.11543math.OC
收藏

熵正则化的随机对策

Entropy-Regularized Stochastic Games

Yagiz Savas, Mohamadreza Ahmadi, Takashi Tanaka, Ufuk Topcu

摘要:

在二人零和随机对策中,两个竞争的局中人在不确定性条件下进行决策,传统上一对最优策略由纳什均衡描述,并在局中人对环境的随机转移模型具有完全信息的假设下计算。然而,实施这种战略可能会使参与者容易受到环境中不可预见的变化的影响。在本文中,我们引入了熵正则化的随机对策,其中每个玩家的目标是最大化的因果熵的战略,除了其预期的回报。正则化项平衡了每个参与者的理性与其对过渡模型错误信息水平的信念。我们考虑熵正则化的$N$阶段和熵正则化的折扣随机对策,并建立在这两个游戏中的值的存在。此外,我们证明了充分的马尔可夫和平稳的混合策略,以达到价值,分别在$N$阶段和折扣游戏。最后,我们提出了算法,这是基于凸优化问题,计算的最优策略。在一个数值例子中,我们演示了所提出的方法的运动规划方案,并说明了预期收益的正则化项的效果。

英文摘要:

In two-player zero-sum stochastic games, where two competing players make decisions under uncertainty, a pair of optimal strategies is traditionally described by Nash equilibrium and computed under the assumption that the players have perfect information about the stochastic transition model of the environment. However, implementing such strategies may make the players vulnerable to unforeseen changes in the environment. In this paper, we introduce entropy-regularized stochastic games where each player aims to maximize the causal entropy of its strategy in addition to its expected payoff. The regularization term balances each player's rationality with its belief about the level of misinformation about the transition model. We consider both entropy-regularized $N$-stage and entropy-regularized discounted stochastic games, and establish the existence of a value in both games. Moreover, we prove the sufficiency of Markovian and stationary mixed strategies to attain the value, respectively, in $N$-stage and discounted games. Finally, we present algorithms, which are based on convex optimization problems, to compute the optimal strategies. In a numerical example, we demonstrate the proposed method on a motion planning scenario and illustrate the effect of the regularization term on the expected payoff.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑