ODICE：通过正交梯度更新揭示分布校正估计的奥秘
ODICE: Revealing the Mystery of Distribution Correction Estimation via Orthogonal-gradient Update
AI总结：
针对离线RL与IL中DICE方法因前后梯度冲突性能不佳的问题，提出将反向梯度投影至前向梯度法平面的正交梯度更新，经理论与实验验证该方法实现SOTA性能与强鲁棒性。
摘要：
在本研究中，我们探究了分布校正估计（DIstribution Correction Estimation, DICE）方法，这是离线强化学习（offline reinforcement learning, RL）和模仿学习（imitation learning, IL）领域的一个重要研究方向。基于DICE的方法会施加状态-动作层面的行为约束，是离线学习的理想选择。然而，它们的性能通常远逊于当前仅采用动作层面行为约束的最优（state-of-the-art, SOTA）方法。在重新审视基于DICE的方法后，我们发现在使用真实梯度更新学习价值函数时存在两个梯度项：前向梯度（针对当前状态计算）和反向梯度（针对下一状态计算）。使用前向梯度与许多离线RL方法高度相似，因此可视为施加动作层面的约束。但如果这两个梯度方向冲突，直接添加反向梯度可能会削弱甚至抵消其效果。为解决这一问题，我们提出了一种简单却有效的改进方案：将反向梯度投影到前向梯度的法平面上，由此得到正交梯度更新——这是一种适用于基于DICE的方法的全新学习规则。我们开展了全面的理论分析，发现投影后的反向梯度会带来状态层面的行为正则化，这也揭开了基于DICE的方法的奥秘：价值学习目标确实试图施加状态-动作层面的约束，但需要以经过校正的方式实现。通过简单示例以及在复杂离线RL和IL任务上的大量实验，我们证明采用正交梯度更新的基于DICE的方法（O-DICE）取得了SOTA性能和极强的鲁棒性。
英文摘要：
In this study, we investigate the DIstribution Correction Estimation (DICE) methods, an important line of work in offline reinforcement learning (RL) and imitation learning (IL). DICE-based methods impose state-action-level behavior constraint, which is an ideal choice for offline learning. However, they typically perform much worse than current state-of-the-art (SOTA) methods that solely use action-level behavior constraint. After revisiting DICE-based methods, we find there exist two gradient terms when learning the value function using true-gradient update: forward gradient (taken on the current state) and backward gradient (taken on the next state). Using forward gradient bears a large similarity to many offline RL methods, and thus can be regarded as applying action-level constraint. However, directly adding the backward gradient may degenerate or cancel out its effect if these two gradients have conflicting directions. To resolve this issue, we propose a simple yet effective modification that projects the backward gradient onto the normal plane of the forward gradient, resulting in an orthogonal-gradient update, a new learning rule for DICE-based methods. We conduct thorough theoretical analyses and find that the projected backward gradient brings state-level behavior regularization, which reveals the mystery of DICE-based methods: the value learning objective does try to impose state-action-level constraint, but needs to be used in a corrected way. Through toy examples and extensive experiments on complex offline RL and IL tasks, we demonstrate that DICE-based methods using orthogonal-gradient updates (O-DICE) achieve SOTA performance and great robustness.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
