互信息正则化的离线强化学习
Mutual Information Regularized Offline Reinforcement Learning
AI总结：
本文提出 MISA 框架，通过状态与动作的互信息下界约束离线 RL 的策略改进方向，并统一 CQL 与 TD3+BC，在 D4RL 上显著优于基线。
摘要：
离线强化学习（offline RL）的主要挑战在于查询分布外动作时出现的分布偏移，这会使策略改进方向受到外推误差的偏差影响。大多数现有方法通过在策略改进或评估期间惩罚偏离行为策略的策略或价值来解决该问题。在本工作中，我们提出了一种新颖的 MISA 框架，从数据集中状态与动作之间互信息（Mutual Information）的角度来处理离线 RL，通过直接约束策略改进方向实现。MISA 构建了由策略和 Q 值参数化的互信息下界。我们表明，优化该下界等价于在离线数据集上最大化单步改进策略的似然。因此，我们将策略改进方向约束在数据流形内。所得算法通过加入互信息正则化，同时增强策略评估和策略改进。MISA 是一个通用框架，将保守 Q 学习（CQL）和行为正则化方法（例如 TD3+BC）统一为特例。我们介绍了 MISA 的 3 种不同变体，并通过实验证明，更紧的互信息下界会带来更好的离线 RL 性能。此外，大量实验表明，MISA 在 D4RL 基准测试的各类任务上显著优于广泛的基线方法，例如在 gym-locomotion 任务上取得 742.9 的总分。我们的代码可在 https://github.com/sail-sg/MISA 获取。
英文摘要：
The major challenge of offline RL is the distribution shift that appears when out-of-distribution actions are queried, which makes the policy improvement direction biased by extrapolation errors. Most existing methods address this problem by penalizing the policy or value for deviating from the behavior policy during policy improvement or evaluation. In this work, we propose a novel MISA framework to approach offline RL from the perspective of Mutual Information between States and Actions in the dataset by directly constraining the policy improvement direction. MISA constructs lower bounds of mutual information parameterized by the policy and Q-values. We show that optimizing this lower bound is equivalent to maximizing the likelihood of a one-step improved policy on the offline dataset. Hence, we constrain the policy improvement direction to lie in the data manifold. The resulting algorithm simultaneously augments the policy evaluation and improvement by adding mutual information regularizations. MISA is a general framework that unifies conservative Q-learning (CQL) and behavior regularization methods (e.g., TD3+BC) as special cases. We introduce 3 different variants of MISA, and empirically demonstrate that tighter mutual information lower bound gives better offline RL performance. In addition, our extensive experiments show MISA significantly outperforms a wide range of baselines on various tasks of the D4RL benchmark,e.g., achieving 742.9 total points on gym-locomotion tasks. Our code is available at https://github.com/sail-sg/MISA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
