PIQL: 用于离线强化学习的投影隐式Q学习与支持约束
PIQL: Projective Implicit Q-Learning with Support Constraint for Offline Reinforcement Learning
AI总结：
PIQL通过引入投影和支持约束改进隐式Q学习，提升离线强化学习的适应性和性能，实现多步价值估计和更严格的有利动作标准，在多个基准上取得最佳效果。
摘要：
离线强化学习（RL）面临由于分布外（OOD）动作导致的外推误差根本性挑战。隐式Q学习（IQL）利用期望回归实现样本内学习。然而，IQL依赖于固定的期望回归超参数和基于密度的策略改进方法，这两种方法都限制了其适应性和性能。在本文中，我们提出了一种PIQL，即一种增强了支持约束的IQL投影变种。在策略评估阶段，PIQL将固定的期望回归超参数替换为基于投影的参数，并将一步价值估计扩展为多步公式。在策略改进阶段，PIQL采用支持约束代替密度约束，确保与策略评估更紧密地对齐。理论上，我们证明PIQL保持了期望回归和样本内学习框架，保证了单调的策略改进，并引入了一个逐步更加严格的有利动作标准。在D4RL和NeoRL2基准上的实验表明，PIQL在多样化的领域中实现了稳健的提升，整体上达到了最先进的性能。
英文摘要：
Offline Reinforcement Learning (RL) faces a fundamental challenge of extrapolation errors caused by out-of-distribution (OOD) actions. Implicit Q-Learning (IQL) employs expectile regression to achieve in-sample learning. Nevertheless, IQL relies on a fixed expectile hyperparameter and a density-based policy improvement method, both of which impede its adaptability and performance. In this paper, we propose Projective IQL (PIQL), a projective variant of IQL enhanced with a support constraint. In the policy evaluation stage, PIQL substitutes the fixed expectile hyperparameter with a projection-based parameter and extends the one-step value estimation to a multi-step formulation. In the policy improvement stage, PIQL adopts a support constraint instead of a density constraint, ensuring closer alignment with the policy evaluation. Theoretically, we demonstrate that PIQL maintains the expectile regression and in-sample learning framework, guarantees monotonic policy improvement, and introduces a progressively more rigorous criterion for advantageous actions. Experiments on D4RL and NeoRL2 benchmarks demonstrate robust gains across diverse domains, achieving state-of-the-art performance overall.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
