一般环境下基于混合贝叶斯的自寻优和Pareto最优策略
Self-Optimizing and Pareto-Optimal Policies in General Environments based on Bayes-Mixtures
摘要:
研究了在未知概率环境下的序贯决策问题。在循环$t$中,动作$y_t$导致感知$x_t$和奖励$r_t$,其中所有量通常可以取决于完整的历史。感知$x_t$和奖励$r_t$从(反应性)环境概率分布$μ$中采样。这种非常一般的设置包括但不限于(部分可观察的,k阶)马尔可夫决策过程。序贯决策理论告诉我们,如果$μ$是已知的,我们应该如何行动,以最大化总的预期回报,即价值。如果$μ$未知,则通常使用强化学习。在贝叶斯方法中,将混合分布$ξ$定义为分布$ν\in\M$的加权和,其中$\M$是包括真实环境$μ$的任何分布类。我们表明,贝叶斯最优策略$p^ξ$的基础上的混合$ξ$是自优化的意义上说,所有$μ\in\M$的平均值渐近收敛到(不可行的)贝叶斯最优策略$p^μ$,它知道$μ$提前实现的最优值。我们表明,$\M$承认自我优化的政策在所有的必要条件,也是充分的。未对$\M$进行其他结构假设。作为一个例子的应用程序,我们讨论遍历马尔可夫决策过程,允许自我优化的政策。此外,我们表明,$p^ξ$是帕累托最优的意义上说,没有其他政策产生更高或相等的价值在{ \em所有}环境$ν\in\M$和严格更高的价值在至少一个。
英文摘要:
The problem of making sequential decisions in unknown probabilistic environments is studied. In cycle $t$ action $y_t$ results in perception $x_t$ and reward $r_t$, where all quantities in general may depend on the complete history. The perception $x_t$ and reward $r_t$ are sampled from the (reactive) environmental probability distribution $μ$. This very general setting includes, but is not limited to, (partial observable, k-th order) Markov decision processes. Sequential decision theory tells us how to act in order to maximize the total expected reward, called value, if $μ$ is known. Reinforcement learning is usually used if $μ$ is unknown. In the Bayesian approach one defines a mixture distribution $ξ$ as a weighted sum of distributions $ν\in\M$, where $\M$ is any class of distributions including the true environment $μ$. We show that the Bayes-optimal policy $p^ξ$ based on the mixture $ξ$ is self-optimizing in the sense that the average value converges asymptotically for all $μ\in\M$ to the optimal value achieved by the (infeasible) Bayes-optimal policy $p^μ$ which knows $μ$ in advance. We show that the necessary condition that $\M$ admits self-optimizing policies at all, is also sufficient. No other structural assumptions are made on $\M$. As an example application, we discuss ergodic Markov decision processes, which allow for self-optimizing policies. Furthermore, we show that $p^ξ$ is Pareto-optimal in the sense that there is no other policy yielding higher or equal value in {\em all} environments $ν\in\M$ and a strictly higher value in at least one.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。