马尔可夫决策过程中信息论不透明性的策略梯度方法
Policy Gradient Methods for Information-Theoretic Opacity in Markov Decision Processes
AI总结:
本文提出一种信息论不透明性度量,在满足任务约束下通过原始对偶梯度算法计算最大不透明马尔可夫策略,并利用隐马尔可夫模型可观测算子计算条件熵梯度。
摘要:
不透明性,或非干扰性,是一种确保外部观察者无法从系统观测中推断出机密信息(即“秘密”)的属性。我们引入了一种信息论不透明性度量,该度量在建模为马尔可夫决策过程(MDP)的系统中,利用给定观察者部分观测条件下的秘密的条件熵来量化信息泄漏。我们的目标是找到一种在不透明性最大化的同时满足任务性能约束的控制策略,假设知情观察者了解控制策略和系统动态。具体而言,我们考虑了一类称为基于状态的不透明性,其中秘密是关于系统过去或当前状态的命题公式;以及基于状态的不透明性的一个特例,称为基于语言的不透明性,其中秘密由时序逻辑公式(LTL)或有限状态自动机识别的正则语言定义。首先,我们证明了在优化信息论不透明性方面,有限记忆策略优于马尔可夫策略。其次,我们开发了一种算法,使用基于原始对偶梯度的算法计算最大不透明马尔可夫策略,并证明了其收敛性。由于不透明性无法表示为累积成本,我们开发了一种新方法,利用隐马尔可夫模型中的可观测算子计算条件熵关于策略参数的梯度。实验结果验证了我们提出方法的有效性和最优性。
英文摘要:
Opacity, or non-interference, is a property ensuring that an external observer cannot infer confidential information (the "secret") from system observations. We introduce an information-theoretic measure of opacity, which quantifies information leakage using the conditional entropy of the secret given the observer's partial observations in a system modeled as a Markov decision process (MDP). Our objective is to find a control policy that maximizes opacity while satisfying task performance constraints, assuming that an informed observer is aware of the control policy and system dynamics. Specifically, we consider a class of opacity called state-based opacity, where the secret is a propositional formula about the past or current state of the system, and a special case of state-based opacity called language-based opacity, where the secret is defined by a temporal logic formula (LTL) or a regular language recognized by a finite-state automaton. First, we prove that finite-memory policies can outperform Markov policies in optimizing information-theoretic opacity. Second, we develop an algorithm to compute a maximally opaque Markov policy using a primal-dual gradient-based algorithm, and prove its convergence. Since opacity cannot be expressed as a cumulative cost, we develop a novel method to compute the gradient of conditional entropy with respect to policy parameters using observable operators in hidden Markov models. The experimental results validate the effectiveness and optimality of our proposed methods.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。