迈向最优策略改进

中文摘要 AI 辅助

实用的强化学习（RL）算法通过在近似评估存在的情况下进行迭代策略改进来学习解决马尔可夫决策过程（MDP）。我们从基本原理出发研究策略改进，将最优策略改进定义为在指定约束下通过单次更新可获得的最佳策略。我们证明，在状态集合上限制的最优改进等价于求解一个诱导的MDP，从而将具有显式或隐式模型的规划刻画为通向最优策略改进的路径。由于实用方法通常通过贪心化形式的迭代改进来求解此类诱导问题，我们朝着在近似评估这一核心实际约束下的最优贪心化迈出步伐。我们将此约束下的贪心化表述为不确定性下的概率决策，并推导出一个对由此产生的目标函数而言最优的新算子。实验上，该算子及其基于梯度的实用近似在GumbelAlphaZero、SAC、ReBRAC和广义策略迭代中提升了总体性能，实验涵盖离散与连续动作、基于模型与无模型、在线与离线强化学习。

英文摘要

Practical Reinforcement Learning (RL) algorithms learn to solve Markov Decision Processes (MDPs) through iterative policy improvement in the presence of approximate evaluation. We study policy improvement from first principles, defining optimal policy improvement as producing the best policy attainable in a single update under specified constraints. We show that optimal improvement restricted to a set of states is equivalent to solving an induced MDP, characterizing planning with an explicit or implicit model as a path towards optimal policy improvement. Because practical methods commonly solve such induced problems through iterative improvement in the form of greedification, we take steps towards optimal greedification under the central practical constraint of approximate evaluation. We formulate greedification under this constraint as probabilistic decision-making under uncertainty and derive a novel operator that is optimal with respect to the resulting objective. Empirically, the operator and its practical gradient-based approximations improve aggregate performance across GumbelAlphaZero, SAC, ReBRAC and Generalized Policy Iteration, in experiments spanning discrete and continuous actions, model-based and model-free, online and offline RL.

发表机构

• TU Delft（代尔夫特理工大学）
• Centrum Wiskunde & Informatica, Amsterdam（阿姆斯特丹数学与计算机科学中心）
• ETH Zürich（苏黎世联邦理工学院）
• University of Alberta（阿尔伯塔大学）
• TU Eindhoven（埃因霍温理工大学）

机构由 AI 辅助整理，请以论文原文为准。