线性占用同态用于折扣马尔可夫决策过程

中文摘要 AI 辅助

我们引入了线性占用同态（LOHs），这是一类马尔可夫决策过程（MDP）变换，它在MDP的可行占用度量空间上定义了MDP同态。与经典的MDP同态框架（通过状态和状态-动作空间上的映射定义）不同，LOHs在占用度量空间之间进行映射，并可表示为线性变换矩阵，这些矩阵揭示了同态本身的重要结构性质。对于这类MDP变换，我们推导了有限维线性不等式约束，用以验证候选变换矩阵是否为有效的LOH，并推导出相应的变换后MDP。我们进一步建立了LOHs、软MDP同态和因子矩阵鲁棒MDP之间的联系。当线性变换矩阵可逆时，我们证明原始MDP在同态变换后等价于一个受约束的MDP，并推导出LOH保持最优值的必要且充分条件。我们在两种设置中展示了LOHs的优势：通过将非矩形参数不确定性集变换为同态矩形MDP来改进鲁棒MDP解，以及将大型MDP分解为较小的独立MDP。

英文摘要

We introduce linear occupancy homomorphisms (LOHs), a class of Markov decision process (MDP) transformations that defines the MDP homomorphism over the feasible occupancy measure space of MDPs. In contrast to classic MDP homomorphism frameworks, which are defined through mappings over the state and state-action spaces, LOHs map between occupancy measure spaces and are representable as linear transformation matrices that expose important structural properties of the homomorphism itself. For this class of MDP transformations, we derive finite dimensional linear inequality constraints that certify when a candidate transformation matrix is a valid LOH and derive the corresponding transformed MDP. We further establish a connection between LOHs, soft MDP homomorphisms, and factored matrix robust MDPs. When the linear transformation matrix is invertible, we prove that the original MDP is equivalent to a constrained MDP after the homomorphic transformation, and derive necessary and sufficient conditions under which the LOH preserves the optimal value. We illustrate the benefits of LOHs in two settings: improving robust MDP solutions by transforming nonrectangular parameter uncertainty sets into homomorphic rectangular MDPs, and decomposing a large MDP into smaller independent MDPs.

发表机构

• Daniel Guggenheim School of Aerospace Engineering, Georgia Institute of Technology（佐治亚理工学院丹尼尔·古根海姆航空航天学院）

机构由 AI 辅助整理，请以论文原文为准。