IDQL：将隐式 Q-Learning 作为具有扩散策略的 Actor-Critic 方法
IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies
AI总结：
本文将隐式 Q-Learning 重新解释为 actor-critic 方法，并提出 IDQL，利用扩散参数化行为策略进行重要性采样提取策略，有效处理分布外动作，在离线强化学习中表现优异且对超参数鲁棒。
摘要：
有效的离线强化学习方法需要妥善处理分布外动作。隐式 Q-Learning（IQL）通过使用修改后的 Bellman backup 仅利用数据集动作训练 Q 函数来解决此问题。然而，尚不清楚究竟哪种策略实际达到了这种隐式训练的 Q 函数所表示的值。在本文中，我们通过泛化 critic 目标并将其与行为正则化的隐式 actor 相关联，将 IQL 重新解释为一种 actor-critic 方法。这种泛化表明了诱导的 actor 如何平衡奖励最大化和与行为策略的散度，其中特定的损失选择决定了这种权衡的性质。值得注意的是，这个 actor 可以表现出复杂和多模态的特征，这表明先前方法中使用的带有 advantage weighted regression（AWR）的条件高斯 actor 存在问题。相反，我们提议使用来自扩散参数化行为策略的样本和从 critic 计算的权重，然后进行重要性采样以得到我们的目标策略。我们引入了 Implicit Diffusion Q-Learning（IDQL），将我们的通用 IQL critic 与该策略提取方法相结合。IDQL 保持了 IQL 易于实现的特点，同时优于先前的离线强化学习方法，并展示了对超参数的鲁棒性。代码可在 https://github.com/philippe-eecs/IDQL 获取。
英文摘要：
Effective offline RL methods require properly handling out-of-distribution actions. Implicit Q-learning (IQL) addresses this by training a Q-function using only dataset actions through a modified Bellman backup. However, it is unclear which policy actually attains the values represented by this implicitly trained Q-function. In this paper, we reinterpret IQL as an actor-critic method by generalizing the critic objective and connecting it to a behavior-regularized implicit actor. This generalization shows how the induced actor balances reward maximization and divergence from the behavior policy, with the specific loss choice determining the nature of this tradeoff. Notably, this actor can exhibit complex and multimodal characteristics, suggesting issues with the conditional Gaussian actor fit with advantage weighted regression (AWR) used in prior methods. Instead, we propose using samples from a diffusion parameterized behavior policy and weights computed from the critic to then importance sampled our intended policy. We introduce Implicit Diffusion Q-learning (IDQL), combining our general IQL critic with the policy extraction method. IDQL maintains the ease of implementation of IQL while outperforming prior offline RL methods and demonstrating robustness to hyperparameters. Code is available at https://github.com/philippe-eecs/IDQL.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
