强化学习的最优判别器加权模仿视角
An Optimal Discriminator Weighted Imitation Perspective for Reinforcement Learning
AI总结：
针对现有对偶强化学习（Dual-RL）无法正确估计最优访问分布比率的问题，提出迭代对偶强化学习（IDRL），通过迭代校正逼近最优比率并优化数据集，在D4RL等数据集上性能和稳定性均优于主流基线。
摘要：
我们提出迭代对偶强化学习（IDRL），这是一种采用最优判别器加权模仿视角解决强化学习（RL）问题的新方法。我们的方法源于一项简单实验：使用离线数据集加额外专家数据集训练判别器，再执行判别器加权行为克隆，在各类数据集上取得了优异结果。该最优判别器权重与对偶强化学习（Dual-RL）中学习到的访问分布比率非常相似，但我们发现现有Dual-RL方法无法正确估计该比率。在IDRL中，我们提出一种校正方法，在无额外专家数据集的情况下迭代逼近离线数据集中的最优访问分布比率。每次迭代中，IDRL利用前一次迭代学到的比率移除零权重的次优转移，并在剩余子数据集上运行Dual-RL。这可视为用前一次迭代的优化访问分布替换行为访问分布，理论上形成了改进访问分布比率的课程，使其更接近最优判别器权重。我们在各类离线数据集（包括D4RL数据集和更真实的受损演示数据）上验证了IDRL的有效性。IDRL在所有数据集上的性能和稳定性均优于强大的Primal-RL和Dual-RL基线。
英文摘要：
We introduce Iterative Dual Reinforcement Learning (IDRL), a new method that takes an optimal discriminator-weighted imitation view of solving RL. Our method is motivated by a simple experiment in which we find training a discriminator using the offline dataset plus an additional expert dataset and then performing discriminator-weighted behavior cloning gives strong results on various types of datasets. That optimal discriminator weight is quite similar to the learned visitation distribution ratio in Dual-RL, however, we find that current Dual-RL methods do not correctly estimate that ratio. In IDRL, we propose a correction method to iteratively approach the optimal visitation distribution ratio in the offline dataset given no addtional expert dataset. During each iteration, IDRL removes zero-weight suboptimal transitions using the learned ratio from the previous iteration and runs Dual-RL on the remaining subdataset. This can be seen as replacing the behavior visitation distribution with the optimized visitation distribution from the previous iteration, which theoretically gives a curriculum of improved visitation distribution ratios that are closer to the optimal discriminator weight. We verify the effectiveness of IDRL on various kinds of offline datasets, including D4RL datasets and more realistic corrupted demonstrations. IDRL beats strong Primal-RL and Dual-RL baselines in terms of both performance and stability, on all datasets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
