AlignIQL：通过约束优化实现隐式Q学习中的策略对齐
AlignIQL: Policy Alignment in Implicit Q-Learning through Constrained Optimization
AI总结：
本文提出AlignIQL及AlignIQL-hard算法，将隐式策略寻找问题形式化为约束优化问题，在保持IQL简洁性的同时解决了策略提取问题，并在D4RL稀疏奖励任务上显著优于IQL和IDQL。
摘要：
隐式Q学习（IQL）是离线强化学习的一个强基线，它通过分位数回归仅使用数据集动作来学习价值函数。然而，如何从学习到的隐式Q函数中恢复隐式策略，以及为什么IQL能够利用加权回归进行策略提取，目前尚不清楚。IDQL将IQL重新解释为一种演员-评论家方法，并得到了隐式策略的权重，但该权重仅对最优价值函数成立。在本工作中，我们引入了一种不同的方式来解决隐式策略寻找问题（IPF），即把该问题形式化为一个优化问题。基于该优化问题，我们进一步提出了两种实用算法AlignIQL和AlignIQL-hard，它们继承了IQL中演员与评论家解耦的优点，并为IQL为何能使用加权回归进行策略提取提供了见解。与IQL和IDQL相比，我们发现我们的方法保持了IQL的简洁性，并解决了隐式策略寻找问题。在D4RL数据集上的实验结果表明，与其他最先进的离线强化学习方法相比，我们的方法取得了有竞争力或更优的结果。特别是在Antmaze和Adroit等复杂稀疏奖励任务中，我们的方法以显著优势优于IQL和IDQL。
英文摘要：
Implicit Q-learning (IQL) serves as a strong baseline for offline RL, which learns the value function using only dataset actions through quantile regression. However, it is unclear how to recover the implicit policy from the learned implicit Q-function and why IQL can utilize weighted regression for policy extraction. IDQL reinterprets IQL as an actor-critic method and gets weights of implicit policy, however, this weight only holds for the optimal value function. In this work, we introduce a different way to solve the implicit policy-finding problem (IPF) by formulating this problem as an optimization problem. Based on this optimization problem, we further propose two practical algorithms AlignIQL and AlignIQL-hard, which inherit the advantages of decoupling actor from critic in IQL and provide insights into why IQL can use weighted regression for policy extraction. Compared with IQL and IDQL, we find our method keeps the simplicity of IQL and solves the implicit policy-finding problem. Experimental results on D4RL datasets show that our method achieves competitive or superior results compared with other SOTA offline RL methods. Especially in complex sparse reward tasks like Antmaze and Adroit, our method outperforms IQL and IDQL by a significant margin.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
