具有未知变化前后核的马尔可夫核变点检测

在未知转移模型的确定性马尔可夫决策过程中实现高效且安全的探索

Efficient and Safe Exploration in Deterministic Markov Decision Processes with Unknown Transition Models

本文提出了一种安全探索算法，通过利用Lipschitz连续性确保在探索过程中不访问危险状态，该算法在确定性马尔可夫决策过程中提供了确定性的安全保证，并通过模拟导航任务验证了其性能。

Exactly Optimal Quickest Change Detection of Markov Chains

This paper establishes that an exactly optimal rule for Bayesian Quickest Change Detection (QCD) of Markov chains is a threshold test on the no change posterior. We also provide a computationally efficient scalar filter for the no change posterior whose effort is independent of the dimension of the chains. We establish that an (undesirable) weak practical super-martingale phenomenon can be exhibited by the no change posterior when the before and after chains are too close in a relative entropy rate sense. The proposed detector is examined in simulation studies.

Safety Verification of Unknown Dynamical Systems via Gaussian Process Regression

The deployment of autonomous systems that operate in unstructured environments necessitates algorithms to verify their safety. This can be challenging due to, e.g., black-box components in the control software, or undermodelled dynamics that prevent model-based verification. We present a novel verification framework for an unknown dynamical system from a given set of noisy observations of the dynamics. Using Gaussian processes trained on this data set, the framework abstracts the system as an uncertain Markov process with discrete states defined over the safe set. The transition bounds of the …

参数化马尔可夫决策过程中带无信息动作的汤普森采样

Thompson Sampling for Parameterized Markov Decision Processes with Uninformative Actions

针对参数化马尔可夫决策过程中存在无信息动作的问题，提出一组假设，在此条件下汤普森采样达到渐近最优遗憾界$O(T^{-1})$，适用于排队、库存控制和动态定价等场景。

Bayesian Risk Markov Decision Processes

We consider finite-horizon Markov Decision Processes where parameters, such as transition probabilities, are unknown and estimated from data. The popular distributionally robust approach to addressing the parameter uncertainty can sometimes be overly conservative. In this paper, we propose a new formulation, Bayesian risk Markov Decision Process (BR-MDP), to address parameter uncertainty in MDPs, where a risk functional is applied in nested form to the expected total cost with respect to the Bayesian posterior distribution of the unknown parameters. The proposed formulation provides more flexi…

Satisfiability Bounds for $ω$-Regular Properties in Bounded-Parameter Markov Decision Processes

We consider the problem of computing minimum and maximum probabilities of satisfying an $ω$-regular property in a bounded-parameter Markov decision process (BMDP). BMDP arise from Markov decision processes (MDP) by allowing for uncertainty on the transition probabilities in the form of intervals where the actual probabilities are unknown. $ω$-regular languages form a large class of properties, expressible as, e.g., Rabin or parity automata, encompassing rich specifications such as linear temporal logic. In a BMDP the probability to satisfy the property depends on the unknown transitions probab…