arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2212.08712cs.AIcs.SYeess.SY
收藏

马尔可夫决策过程的因果时序推理

Causal Temporal Reasoning for Markov Decision Processes

Department of Informatics, King's College London(信息学院,伦敦国王学院)

Milad Kazemi, Nicola Paoletti

AI总结:

本文提出PCFTL,一种支持干预与反事实查询的概率时序逻辑,通过结构因果模型解释MDP,并在安全强化学习基准中验证其有效性。

摘要:

我们引入PCFTL(概率反事实时序逻辑),这是一种用于验证马尔可夫决策过程(MDP)的新型概率时序逻辑。PCFTL首次包含因果推理算子,使我们能够表达干预性和反事实查询。给定路径公式ϕ,干预性质关注的是如果我们对MDP施加特定改变I(例如切换到不同策略),ϕ的满足概率;反事实则允许我们在给定观测到的MDP路径τ的情况下,计算如果我们过去施加了I,ϕ的结果会是什么。由于能够推理涉及MDP不同配置的“假设”场景,我们的方法区别于现有的只能推理固定系统配置的概率时序逻辑。从语法角度看,我们引入了一个广义反事实算子,它涵盖了干预概率和反事实概率,以及例如PCTL中的传统概率算子。从语义角度看,我们的逻辑在MDP的结构因果模型转换上进行解释,这为我们提供了一种适合反事实推理的表示。我们使用网格世界模型基准在安全强化学习的背景下评估PCFTL。

英文摘要:

We introduce $\textit{PCFTL (Probabilistic CounterFactual Temporal Logic)}$, a new probabilistic temporal logic for the verification of Markov Decision Processes (MDP). PCFTL is the first to include operators for causal reasoning, allowing us to express interventional and counterfactual queries. Given a path formula $ϕ$, an interventional property is concerned with the satisfaction probability of $ϕ$ if we apply a particular change $I$ to the MDP (e.g., switching to a different policy); a counterfactual allows us to compute, given an observed MDP path $τ$, what the outcome of $ϕ$ would have been had we applied $I$ in the past. For its ability to reason about \textit{what-if} scenarios involving different configurations of the MDP, our approach represents a departure from existing probabilistic temporal logics that can only reason about a fixed system configuration. From a syntactic viewpoint, we introduce a generalized counterfactual operator that subsumes both interventional and counterfactual probabilities as well as the traditional probabilistic operator found in e.g., PCTL. From a semantics viewpoint, our logic is interpreted over a structural causal model translation of the MDP, which gives us a representation amenable to counterfactual reasoning. We evaluate PCFTL in the context of safe reinforcement learning using a benchmark of grid-world models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑