重新审视鲁棒马尔可夫决策过程中的次梯度主导性:反例、难度和充分条件
Revisiting Subgradient Dominance in Robust MDPs: Counterexamples, Hardness, and Sufficient Conditions
AI总结:
本文指出鲁棒马尔可夫决策过程在一般紧不确定性集下不满足次梯度主导性,且寻找ε最优策略可能为NP难问题,但提出两种条件保证次梯度主导性。
摘要:
投影次梯度下降(PSD)因适用于更广泛的不确定性集而被用于求解鲁棒马尔可夫决策过程(RMDPs)。现有研究声称,具有通用紧不确定性集的RMDPs满足次梯度主导性,从而在多项式次迭代中收敛到ε最优策略(如Wang等,2023)。我们证明这些主张不正确。即使当不确定性集的基数为二时,RMDP目标也不满足次梯度主导性,并可能允许次优的严格局部极小值。此外,我们证明在子梯度可高效计算的设定下,寻找ε最优策略可能是NP难的:(i) 有限转移不确定性集;(ii) 有限成本不确定性集的sa-矩形有限转移不确定性集。最后,我们识别出两种条件,使得RMDPs确实满足次梯度主导性:当每个策略的最坏转移核或最坏动作值函数唯一时。
英文摘要:
Projected subgradient descent (PSD) has gained popularity for solving robust Markov decision processes (RMDPs) because it applies to a broader class of uncertainty sets than traditional dynamic programming. Existing work claims that RMDPs with a general compact uncertainty set satisfy the subgradient dominance property, under which exact PSD finds an $\varepsilon$-optimal policy in a polynomial number of updates (e.g., Wang et al., 2023). We show that these claims are incorrect. Even when the uncertainty set has cardinality two, the RMDP objective need not be subgradient-dominant and can admit suboptimal strict local minima. Moreover, we prove that finding an $\varepsilon$-optimal policy can be NP-hard even in settings where subgradients are efficiently computable: (i) finite transition uncertainty sets and (ii) $sa$-rectangular finite transition uncertainty sets with finite non-rectangular cost uncertainty sets. Finally, we identify two conditions under which RMDPs do satisfy subgradient dominance: when, for each policy, either the worst-case transition kernel is unique or a robust action-value function exists.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。