基于差Q函数正交估计的决策中心抽象
Decision-Centered Abstractions via Orthogonal Estimation of Difference-of-Q Functions
AI总结：
本文提出通过正交估计差Q函数学习决策中心状态抽象，在离线强化学习中简化状态空间，提升策略优化效率与收敛性，并减少方差。
摘要：
离线强化学习使得在无法因安全、成本及其他顾虑而在线部署新策略时，能够从历史数据中评估和优化序贯决策。大数据进展提供了丰富的状态信息，但可能天真地包含与奖励和动作无关的动态，这些动态对于学习最优动作最终是不必要的。我们引入了旨在保留差Q函数的状态抽象，并提议通过差Q函数的因果机器学习与标准统计稀疏学习来学习这些抽象。在非参数加性奖励模型下，我们刻画了决策中心抽象何时比完整状态空间更简单，从而激励了我们的估计过程。我们开发了R学习器（Nie等人2021，Lewis和Syrgkanis 2021）的动态推广，用于估计离散值动作a、a0的差Q函数。我们利用正交估计来提高收敛速度，即使所需的Q函数和行为策略估计以较慢速度收敛，并证明在边际条件下策略优化的一致性。该方法可以利用Q函数和行为策略的黑箱估计器来针对更结构化的Q函数对比进行估计，并使用简单的平方损失最小化。我们在模拟数据和模拟器增强的真实数据中展示了我们估计器的方差改进，以及我们的方法如何使我们能够隔离序贯决策所需的信息，这些信息可能少于状态预测所需的信息。
英文摘要：
Offline reinforcement learning enables evaluation and optimization of sequential decisions from historical data, when it is not possible to deploy new policies online due to safety, cost, and other concerns. Big data advances enable rich state information, but may naively include reward- and action- irrelevant dynamics that are ultimately unnecessary for learning optimal actions. We introduce state abstractions that target preservation of the difference-of-Q functions, and we propose to learn these abstractions via causal machine learning of the difference-of-Q function and standard statistical sparse learning. Under a nonparametric additive-rewards model, we characterize when decision-centered abstractions are simpler than the full state space, motivating our estimation procedure. We develop a dynamic generalization of the R learner (Nie et al. 2021, Lewis and Syrgkanis 2021) for estimating difference of Q-functions, for discrete-valued actions a, a0. We leverage orthogonal estimation to improve convergence rates, even if the required estimates of Q and behavior policy converge at slower rates and prove consistency of policy optimization under a margin condition. The method can leverage black-box estimators of the Q-function and behavior policy to target estimation of a more structured Q-function contrast, and uses simple squared-loss minimization. We demonstrate variance improvements from our estimator and how our approach enables us to isolate the information needed for sequential decision-making, which can be less than that for state prediction, in simulated data and simulator-augmented real data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
