在马尔可夫决策过程中的最优控制中采用最快变化检测方法
Quickest Change Detection Approach to Optimal Control in Markov Decision Processes with Model Changes
AI总结：
本文提出了一种基于最快变化检测的方法，用于解决非平稳马尔可夫决策过程中的最优控制问题，通过双阈值切换策略提升长期奖励表现。
摘要：
非平稳马尔可夫决策过程（MDP）中的最优控制是一个具有挑战性的问题。在这样的控制问题中，目标是在过渡动态或奖励函数随时间变化时最大化长期折扣奖励。当已知变化统计信息时，标准的贝叶斯方法是将问题重新表述为部分可观测马尔可夫决策过程（POMDP）并使用近似POMDP求解器解决，这通常计算上很耗费资源。在本文中，从最快变化检测（QCD）的视角分析该问题，QCD是一组用于检测随机变量序列分布变化的工具。当前应用QCD到此类问题的方法仅通过遵循预设策略被动检测变化，而没有优化长期性能的行动选择。本文证明忽略奖励-检测权衡可能导致长期奖励显著损失，并提出双阈值切换策略来解决这一问题。还提出了一个非贝叶斯问题形式化，用于无法定义贝叶斯形式化的场景。通过在非平稳MDP任务上的数值分析检验所提双阈值策略的性能，该策略在贝叶斯和非贝叶斯设置中均优于现有最先进QCD方法。
英文摘要：
Optimal control in non-stationary Markov decision processes (MDP) is a challenging problem. The aim in such a control problem is to maximize the long-term discounted reward when the transition dynamics or the reward function can change over time. When a prior knowledge of change statistics is available, the standard Bayesian approach to this problem is to reformulate it as a partially observable MDP (POMDP) and solve it using approximate POMDP solvers, which are typically computationally demanding. In this paper, the problem is analyzed through the viewpoint of quickest change detection (QCD), a set of tools for detecting a change in the distribution of a sequence of random variables. Current methods applying QCD to such problems only passively detect changes by following prescribed policies, without optimizing the choice of actions for long term performance. We demonstrate that ignoring the reward-detection trade-off can cause a significant loss in long term rewards, and propose a two threshold switching strategy to solve the issue. A non-Bayesian problem formulation is also proposed for scenarios where a Bayesian formulation cannot be defined. The performance of the proposed two threshold strategy is examined through numerical analysis on a non-stationary MDP task, and the strategy outperforms the state-of-the-art QCD methods in both Bayesian and non-Bayesian settings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
