arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2312.08536cs.LGcs.SYeess.SY
收藏

带噪声状态观测的马尔可夫决策过程

Markov Decision Processes with Noisy State Observation

Stanford University(斯坦福大学)

Amirhossein Afsharrad, Sanjay Lall

AI总结:

针对马尔可夫决策过程中的噪声状态观测问题,提出二阶重复动作法和贝叶斯算法族两类噪声估计方法,经仿真验证有效,推进了噪声环境下强化学习的状态估计技术。

摘要:

本文解决了马尔可夫决策过程(Markov Decision Processes, MDPs)中一类特定的带噪声状态观测难题,这是各类现实应用中的常见问题。我们聚焦于通过一个混淆矩阵对这种不确定性建模,该矩阵捕获了误识别真实状态的概率。我们的主要目标是估计固有测量噪声,为此我们提出了两种新颖的算法方法。第一种是二阶重复动作法,旨在有限时间窗口内实现高效的噪声估计,为系统分析提供可识别性条件。第二种方法包含一族贝叶斯算法,我们对其性能与局限性进行了全面分析与比较。我们通过仿真验证了理论发现,证明了所提方法在不同场景下的有效性,尤其突出了它们在具有不同平稳分布的环境中的表现。我们的工作推进了对噪声环境下强化学习的理解,为MDP中更精确的状态估计提供了鲁棒技术。

英文摘要:

This paper addresses the challenge of a particular class of noisy state observations in Markov Decision Processes (MDPs), a common issue in various real-world applications. We focus on modeling this uncertainty through a confusion matrix that captures the probabilities of misidentifying the true state. Our primary goal is to estimate the inherent measurement noise, and to this end, we propose two novel algorithmic approaches. The first, the method of second-order repetitive actions, is designed for efficient noise estimation within a finite time window, providing identifiable conditions for system analysis. The second approach comprises a family of Bayesian algorithms, which we thoroughly analyze and compare in terms of performance and limitations. We substantiate our theoretical findings with simulations, demonstrating the effectiveness of our methods in different scenarios, particularly highlighting their behavior in environments with varying stationary distributions. Our work advances the understanding of reinforcement learning in noisy environments, offering robust techniques for more accurate state estimation in MDPs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑