随机最短路径问题的贝叶斯学习
Bayesian learning for the stochastic shortest path problem
AI总结:
针对随机最短路径问题,提出一种贝叶斯框架,通过贝尔曼最优方程直接构建最优动作价值函数Q*的后验分布,并解决似然松弛导致的不可识别性问题,实现不确定性量化与数据高效学习。
摘要:
序列决策问题通常被建模为马尔可夫决策过程(MDP)。我们关注随机最短路径(SSP)问题,这是一个具有吸收终止状态的无限水平无折扣MDP。我们开发了一个贝叶斯框架,通过与决策任务的交互来学习最优决策策略。具体来说,我们学习最优动作价值函数$Q^*$,但与许多现有的贝叶斯方法不同,我们不依赖于不现实的建模假设和临时近似。我们的方法是通过贝尔曼最优方程直接构建$Q^*$的后验信念。对于确定性奖励,我们将后验描述为具有流形密度的分布。为了简化推理,我们放松了似然,使得勒贝格密度存在。但这样做的代价是产生不可识别性问题。具体来说,放松后的后验可能在不当决策规则上有显著质量,而精确后验则不会。我们还计算了$Q^*$的表格参数化、高斯似然放松和高斯先验下最优动作选择的精确后验概率,这在基准测试研究中很有用。对深海基准测试变体的数值研究验证了我们的发现。我们证明了我们的框架能够忠实地量化不确定性,并且与其他基于时间差分的贝叶斯方法相比,数据效率更高。最后,我们对未来工作提出了建议。
英文摘要:
Sequential decision-making problems are often modelled as a Markov decision process (MDP). We focus on the stochastic shortest path (SSP) problem, which is an infinite-horizon undiscounted MDP with absorbing terminal states. We develop a Bayesian framework to learn the optimal decision strategy through interactions with the decision-making task. Specifically, we learn the optimal action-value function $Q^*$, but unlike many existing Bayesian approaches, we do not rely on unrealistic modelling assumptions and ad-hoc approximations. Our approach is to directly construct the posterior beliefs for $Q^*$ through Bellman's optimality equations. For deterministic rewards, we characterise the posterior as a distribution with a manifold density. To facilitate simpler inference, we relax the likelihood so that a Lebesgue density exists. The flip side is to create unidentifiability issues. Specifically, the relaxed posterior can have significant mass on improper decision rules, while the exact posterior will not. We also calculate the exact posterior probabilities for optimal action selections for the tabular parametrisation of $Q^*$, a Gaussian likelihood relaxation and a Gaussian prior, which is useful in benchmarking studies. Numerical studies on variants of the Deep Sea benchmark verify our findings. We demonstrate that our framework faithfully quantifies uncertainty and, compared to other temporal-difference-based Bayesian methodologies, is more data efficient. We conclude with recommendations for future work.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。