策略学习问题的层级体系
A Hierarchy of Policy Learning Problems
AI总结：
该研究提出策略学习问题的数学框架，定义最优策略、改进策略、策略存在三类问题，证明三者样本复杂度依次降低，即便无法找到改进策略也可判断其是否存在。
摘要：
策略学习旨在从观测数据中学习用于决策的策略，已受到广泛关注。该领域的多数工作聚焦于开发算法来计算与最优策略相比可最小化遗憾的策略。但在诸多实际场景中，没有足够数据来获得低遗憾。因此，近期工作将注意力转向替代目标，最具代表性的是研究是否有可能学习到一个在统计上显著优于基线策略的改进策略。我们认为研究更广泛的策略学习问题具有重要价值。当没有足够数据学习改进策略时，仍可能存在可解答的有用问题。为此，我们提出了一个用于研究各类策略学习问题间关系的数学框架。我们在该框架内形式化了三类问题：除最优策略问题和改进策略问题外，还提出了策略存在问题，其目标是确定是否存在改进策略。在我们的框架内，证明了策略存在问题可归约到改进策略问题，后者又可归约到最优策略问题；这些归约结果证明，每个问题（在样本复杂度层面）至少比后一个问题更容易。一个核心问题仍待解答：这种难度差异是否严格？我们给出了部分答案：首先，最优策略问题与改进策略问题之间的难度差距是严格的。对于改进策略问题和策略存在问题，我们证明当改进策略学习算法满足自然条件时，二者存在亚多项式级的难度差距。因此，即便我们无法找到改进策略，仍有可能解答关于改进策略是否存在的相关问题。这些结果凸显了研究更广泛的策略学习问题的价值。
英文摘要：
Policy learning has received substantial attention with the goal of learning policies from observational data for decision-making. A majority of work in this space has focused on developing algorithms for computing policies that minimize regret compared to the optimal policy. However, in many practical settings, there is insufficient data to obtain low regret. As a result, recent work has shifted attention to alternative objectives, most notably, studying whether it is possible to learn an improving policy that statistically significantly outperforms baseline policies. We argue that there is substantial merit in studying a broader range of policy learning problems. When there is insufficient data to learn an improving policy, there may still be useful questions that can be answered. To this end, we provide a mathematical framework for studying the relationships between policy learning problems. We formalize three problems within our framework: beyond the optimal policy problem and the improving policy problem, we also propose the policy existence problem, which aims to determine if an improving policy exists. Within our framework, we show that the policy existence problem reduces to the improving policy problem, which in turn reduces to the optimal policy problem; these reductions prove that each problem is at least as easy as the next one (in sample complexity). A key question remains: is this hardness strict? We provide partial answers. First, the gap between the optimal policy and improving policy problems is strict. For the improving policy and policy existence problems, we prove that a sublinear polynomial gap exists under natural conditions on improving policy learning algorithms. Thus, we may be able to answer questions about the existence of an improving policy even when we cannot find one. These results highlight the value in studying a broader range of policy learning problems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
