因果流Q学习用于鲁棒的离线强化学习
Causal Flow Q-Learning for Robust Offline Reinforcement Learning
AI总结：
本文提出一种因果流Q学习方法，通过优化策略在混淆偏见下的最坏情况性能，提升离线强化学习的鲁棒性。
摘要：
基于流匹配的表达性策略最近在强化学习（RL）中成功应用，因为它们能够从离线数据中建模复杂的动作分布。这些算法建立在标准策略梯度之上，假设数据中没有未测量的混淆因素。然而，当演示者与学习者的感觉能力存在不匹配时，像素化演示数据中隐含的混淆偏见可能无法满足这一条件。我们通过从因果视角研究离线RL中的混淆观测问题来解决这一挑战。我们开发了一种新的因果离线RL目标，该目标优化策略在可能因混淆偏见而产生的最坏情况下的性能。基于这一新目标，我们引入了一种实际的实现方法，从存在混淆的演示中学习具有表达性的流匹配策略，采用深度判别器来评估目标策略与名义行为策略之间的差异。在25个像素化任务上的实验表明，我们提出的抗混淆增强过程在成功率达到现有最先进离线RL方法的120%。
英文摘要：
Expressive policies based on flow-matching have been successfully applied in reinforcement learning (RL) more recently due to their ability to model complex action distributions from offline data. These algorithms build on standard policy gradients, which assume that there is no unmeasured confounding in the data. However, this condition does not necessarily hold for pixel-based demonstrations when a mismatch exists between the demonstrator's and the learner's sensory capabilities, leading to implicit confounding biases in offline data. We address the challenge by investigating the problem of confounded observations in offline RL from a causal perspective. We develop a novel causal offline RL objective that optimizes policies' worst-case performance that may arise due to confounding biases. Based on this new objective, we introduce a practical implementation that learns expressive flow-matching policies from confounded demonstrations, employing a deep discriminator to assess the discrepancy between the target policy and the nominal behavioral policy. Experiments across 25 pixel-based tasks demonstrate that our proposed confounding-robust augmentation procedure achieves a success rate 120\% that of confounding-unaware, state-of-the-art offline RL methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
