离线强化学习中的百分位准则优化
Percentile Criterion Optimization in Offline Reinforcement Learning
AI总结：
针对离线强化学习中百分位准则非凸、贝叶斯可信区域导致策略过度保守的问题，提出基于Value-at-Risk的动态规划算法，无需显式构建模糊集即可获得更不保守的鲁棒策略。
摘要：
在强化学习中，针对数据有限的高风险决策问题，鲁棒策略通常通过优化百分位准则来计算。百分位准则的近似求解方式是：构建一个以高概率包含真实模型的模糊集，并针对该集合中的最差模型优化策略。由于百分位准则是非凸的，构建模糊集往往具有挑战性。现有工作采用贝叶斯可信区域作为模糊集，但它们通常不必要地过大，并导致学习到过于保守的策略。为克服这些缺陷，我们提出一种新颖的基于风险价值（Value-at-Risk）的动态规划算法，用于优化百分位准则，而无需显式构建任何模糊集。我们的理论和实证结果表明，该算法隐式构建了小得多的模糊集，并学习到保守程度更低的鲁棒策略。
英文摘要：
In reinforcement learning, robust policies for high-stakes decision-making problems with limited data are usually computed by optimizing the \emph{percentile criterion}. The percentile criterion is approximately solved by constructing an \emph{ambiguity set} that contains the true model with high probability and optimizing the policy for the worst model in the set. Since the percentile criterion is non-convex, constructing ambiguity sets is often challenging. Existing work uses \emph{Bayesian credible regions} as ambiguity sets, but they are often unnecessarily large and result in learning overly conservative policies. To overcome these shortcomings, we propose a novel Value-at-Risk based dynamic programming algorithm to optimize the percentile criterion without explicitly constructing any ambiguity sets. Our theoretical and empirical results show that our algorithm implicitly constructs much smaller ambiguity sets and learns less conservative robust policies.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
